Why doesn’t AI have memory like humans do?

Introduction

Mitra:
‍Hi and welcome to a new episode of Syntra. Today we're going to talk about memory. Before we dive in – if you haven't listened to Syntra before: this is the podcast for people driving their function forward who want to understand how the relationship between systems and users shapes the way we work. We share insights on how new technological possibilities are transforming the way we work and interact with technology. Welcome back, Martin and Glenn!

Glenn & Martin:
‍Thanks, thanks.

Mitra:
‍Today we're talking about memory. What do you have in mind? What do we actually mean by memory?

What Do We Mean by Memory?

Glenn:
‍I think it's a fascinating question. What I mean by memory in today's conversation is really agents – how they handle memory, or rather: the agent we interact with day to day has no memory at all. And what are the consequences of that?

Martin:
‍Exactly. It's important to start with some kind of definition. What do we actually mean by memory? It's the context of an LLM – a large language model. I think most people interact day to day with a chat interface, and a chat interface connected to an LLM has essentially no memory. People try to solve it, but it's a diffuse problem. An LLM as such has no memory built in – an LLM, or large language model as we say in Swedish. It has a short-term memory. It doesn't remember anything from a previous session.

Glenn:
‍Right. It can feel like it has very good memory within a conversation, but that's just because the entire conversation – every time you ask a new question or add something – is sent to it in full, including everything said earlier in the same session. Which is actually quite amusing: sending the entire conversation back to its last point. But when you have a long conversation with an AI, there's eventually a lot to send back. And then it becomes easy for an AI to lose track of what's actually relevant to the current conversation. Maybe what we talked about in this conversation two weeks ago? Maybe not relevant, but it still has all the details.

Martin:
‍Right, and if you don't understand that this is how an LLM works, there's also a risk that you won't start a new conversation when you switch topics – when you're talking about very different things within one, sometimes unmanageable, context. It can get very confusing.

Why Memory Matters for Agents

Mitra:
‍But what's driving this conversation now? Why did this feel like a topic we just had to cover?

Glenn:
‍It really comes down to the whole trend of agents gaining ground. If they're actually going to help out in everyday work, they need to know what they've done before, about their surroundings, about their assignment.

Martin:
‍Yes, and I think we've reached a point where AI is smart enough to do almost anything – but it doesn't know what it should do. It doesn't know what you actually need, because it needs to remember a few things about you. People often talk about context – that agents need the right context to solve a problem. That's not where we're stuck. It's specifically the memory question. How is it supposed to know what you need?

Glenn:
‍And it becomes obvious when you think about it: every time I want to use a tool I need to prompt the agent and give it context, and every time I want to use a new tool I need to repeat that process and give the agent the context all over again. If there's no system for that, you end up managing long-term memory manually.

Martin:
‍Right, short-term memory we touched on already – that's fairly easy to solve, you just send the whole conversation. But long-term memory – remembering things between conversations – that's a completely different challenge. You often run into the problem of talking about very different things. Should you really share memory between these two conversations? If one moment I'm talking about updating a product image on a coffee machine and the next about product search, maybe it's not relevant to share memory between those two discussions.

Glenn:
‍Or could it be? Imagine it could spot patterns between them. But unfortunately those might be patterns it shouldn't be seeing. It can mix up memories – it might make an abstraction of something and think "this is super relevant information, let's draw on what we talked about this morning" – but that could lead to a wrong result, just because the model isn't smart enough in that moment to separate those different memories.

How AI Memory Is Handled Today

Martin:
‍The way memory is handled today is really through some kind of external system that retrieves a bit of text about what was discussed before. But you always send the concrete, specific memory – you don't send along concepts or patterns as such.

Glenn:
‍It's a difficult thing to explain, but you just send text – specific memories. That's nothing like how our memory works, not in the way we're used to thinking about memory.

Martin:
‍No, exactly. Memories are connected through different threads that can be very hard to define from the outside. And it's not always logical. I think of a red ball and it makes me think of something completely different – that's how chains of association work. They're not always language-based – episodic memories can be very strongly triggered by smells, visual impressions or situations. That's what we rely on in everyday life, but LLMs have to work within language.

Mitra:
‍But if we continue on the problem side – we've started to define what we mean by memory. What are the challenges you see, particularly with customers, but also as we start using agents more and more?

Glenn:
‍It's a lot about being able to get real value out of these LLMs. You can have a chatbot – you can get some value out of it, but you don't actually save that much time compared to doing the task yourself. But if you have an AI that becomes more agentic and can make its own decisions and act on your behalf, you can save a whole lot more time. That's where AI can actually live up to its promise – given all the money being poured into it. The market believes AI will be able to expand the value it delivers. But to reach that point, we need to free ourselves a little more from hand-holding the AI – not describing everything in minute detail – and that's what memory is supposed to solve.

Selective Memory and the Challenge of Learning

Martin:
‍Exactly, because an AI needs to find the right memory at the right time. If you have a narrow agent doing a specific thing, it might be fairly easy to pinpoint: these are the specific types of memories relevant to that operation. But it also needs to be able to update its memory selectively. "Now I want to learn something new about this thing we keep dealing with, and apply it going forward." If an operation didn't go well, the agent needs to update its memory in a way that doesn't create false beliefs – where it thinks the wrong way of doing something is the right way.

Glenn:
‍All of this comes down to the fact that an AI is stochastic. Take a very simple example of using AI in a business. You use AI to create product descriptions. Simple example. But you need products in the same category to always follow roughly the same pattern. That's very hard to express to an AI, because it can't remember what to follow. We can't even tell an AI to be consistent, because it can't remember patterns. What you can do is send along other example texts and say "this is how I've written before". But then you're sending specific examples. And you, as the person configuring the work, have to decide in advance which examples are relevant here.

Martin:
‍We want AI to consult itself less and say: "Do we know any products we know have received high scores before? Maybe we can pull those as examples." But they might describe something different from what we need right now.

Glenn:
‍Why can't it remember? It can remember specifically – if we implement the right kind of tooling, you can pull in a specific context from something that happened earlier. But then you get a specific memory. You get the exact product text. But not the feeling of it – what you were trying to express, the pattern in it. You can give an AI the whole text as a memory. And if you combine that with the fact that an AI is like an imitation machine – what you put in is what it keeps going in the same direction with – then you get it describing similar products again.

Mitra:
‍If I understand correctly: I can give an AI instructions and a context, and it can remember that specific context. But there's more information I haven't shared – the feeling or the surrounding context of that information. That's something I may never have given the AI, so it's very limited in what it can draw on.

Glenn:
‍There are really two challenges. One is how to store information in a way that's more like human memory. It's a person who reviews AI output and makes a call – this was right or this was wrong. But consolidating that as memory isn't straightforward. You're not necessarily looking for a specific example. If I'm looking at a specific product of the type "nail", it's more of a holistic kind of memory – a screw is fairly similar to a nail and so on. You build a web of how you expect things to relate, which isn't concrete. The second challenge is how AI should formulate queries to a database to retrieve the right thing. Classic database retrieval searches on key-value. You know you have a key and you look for an index. You know exactly what you're after. But if the need is vaguely formulated – more like human memory – how do you describe it when you're fetching from the database? "I'm interested in things that are nail-like." How do you describe "nail-like"?

Is Human Memory the Goal?

Mitra:
‍Is the goal for AI to have a memory like ours – to work the same way we do? Is that what we're aiming for?

Glenn:
‍We humans are the best example we have of memory that works well. But that doesn't necessarily make it the best solution.

Martin:
‍I think we can draw a lot of inspiration from ourselves. We have the superpower of being able to forget things. We remember concepts without remembering details – it's a kind of self-reinforcing system. You remember the good things and there's no real value in everyday life to remembering the bad ones. So you forget what you don't use often. One approach LLM makers are taking to attack this problem is expanding the context window – how much more text you can feed the model, giving it entire books. Then you can give it as much memory as you want. But that creates a complication: you're still not sending a feeling. You're sending a lot of raw information and you don't know what you're looking for in a long text. And probably a lot of irrelevant information the model just has to process, which is costly.

Glenn:
‍And is a feeling what we actually want to send? Or is it really the limitation we've identified? How different should the approach to the memory problem actually be depending on the use case? If you're working on a specific task, maybe memory clusters in a certain way. If you're doing something completely different, maybe it's a different kind of memory. Is there one ultimate perfect solution for memory? Probably not.

Can AI Learn?

Mitra:
‍Something I've heard from several customers is exactly this: "Can't AI just learn?" If we know that a certain error tends to come up in a certain situation – can't it just learn from that? "You know you're dealing with this situation now. Remember the ten previous times this happened. Can you learn from that?" Is that something you can achieve?

Glenn:
‍That's the holy grail. It can be seen as a kind of learning if it works – understanding that it made a mistake and getting better. But the systems we have today genuinely cannot do that, at all. We get asked that question a lot – when will it be able to remember things? But actually learning on top of that is very closely related to memory, and I haven't seen anything that does it well.

Martin:
‍No, theoretically it feels like a manageable task. We have tool-calling in our models. They can perform tasks, interact with a system, retrieve memory from a database and write memory back to a database. But specifically preventing cases where things that are really just hallucinations get stored as real memories – that requires a kind of framework that's probably achievable. But are we there now? It's an area that's being explored enormously. New tools come out every week. But all of them suffer from the same problem: who should own this memory?

Glenn:
‍Because I interact with a whole range of different tools. I want them to share memory in some way – but also not. Some parts of my memory I want to carry between different applications, some I don't. And if you put it on the model or the specific agent – do we choose a global memory that covers everything in every situation? Or is this memory just for this particular LLM session?

How Can Companies Approach Memory Today?

Mitra:
‍How can companies approach memory today? What tools or techniques are available right now that you as a user can use to get more out of AI in your team? Is there anything you can do to influence memory and make things easier?

Glenn:
‍It depends entirely on which tool you're in, which type of agent you use day to day. If you're in Copilot, it doesn't have a memory as such. But it does at least have access to all of your company's documentation. That's one way to manage an organization's memory – being able to search through files. But even then, it's only analogies: it's only specific examples you can search for. You can't get that "feeling" in.

Martin:
‍But really, regardless of which system you're in – there's a trick I use often when jumping between systems: asking the LLM for a memory summary. It summarizes everything from the current conversation – not a built-in memory function. You then have everything in a single format, as a text you can copy over to the next system where you need the content of that conversation. You can essentially consolidate – is that the right word? – a long conversation into a more abstract form. You might get it expressed in three sentences instead of a five-hour discussion. Then you can carry that forward and it acts roughly like an abstract memory. But it's still expressed in text, not as a feeling.

Glenn:
‍But you also avoid having to give the prompt or the context again – you can just take it and use it as an input to another model and paste it in. It improves the loop a little.

Martin:
‍Exactly. But it's still a manual step – you have to ask for it yourself, and you have to carry the context forward to the next situation. The next step might be that you can give a simple instruction to do this exercise every time and pass it on automatically. But you still need someone keeping track of it.

Glenn:
‍The moment you're copy-pasting between different systems, I think we're in an immature situation. We haven't landed on what memory should be. It's a completely open discussion, and if we have the same conversation a year from now, I think there will be some kind of framework in place – a global standard for how memory is handled.

Martin:
‍Exactly. Or several. There might be situations and use cases where one approach works better than another. Or maybe there are multiple types of memory – an agent might have one solution for more global memory shared between agents, and another for just this specific LLM session.

Memory in Practice

Mitra:
‍But if we look at customers in their everyday work – especially the professional groups we work with a lot – if we look into their work processes: what tools are they in when they carry out a task? They're in this tool doing that task. Here you could take the consolidated data and use it as input into another tool. Can we give a concrete example of what that kind of process might look like?

Glenn:
‍Most people with work tasks, I think, start in some kind of case management system or an Excel file received from somewhere, with an assignment. "I need to work with this type of article data, I need to update prices on a few items and publish them to an e-commerce site." A typical task might be getting an Excel file from a supplier. You need to read that file, find a way to import it into the system, transform it so it can be imported – and then it needs to be published to the e-commerce site and validated. But in most processes like this, we haven't introduced AI yet. There's potential to do it, we just haven't quite landed on where exactly it should come in.

Martin:
‍AI is brilliant at reading Excel files and transforming them with tools to get them into the right format. If it's something I use every single day – using AI to convert an Excel file – it would create enormous value if AI learned exactly how I want it to transform that file. That's really where we haven't gotten very far yet – using AI fully in the processes.

Glenn:
‍You can look at developers. They're really the first group to go there, because the results are very testable when you're writing code. It's far ahead. And there I see the memory problems very clearly. Everything when you switch between different projects – I might have my favorite code stack, I always want to code in Python for example – but every time I start a new project I need to describe it again and which functions I want to use in Python, my way of writing things. I need to bring that with me every time. Every time I start a new project, it's a blank memory. We'll see the same problems as we introduce AI in other roles too, not just developers. This is the first testing phase.

Martin:
‍We can feel that if we take a more everyday example: it's when you've had a meeting and not everyone who should have heard the information was there. That leads to a meeting series where you need to have more of the same meeting, because you need to repeat the same context for the next person. That's the closest non-automated analogy I can think of right now.

Glenn:
‍That's a great point. It's the same problem – and you notice there how hard it is to repeat the same information next time. Because you feel like "I already communicated this in the last meeting", but there are new people who didn't hear it. And another phenomenon you can find yourself in is the situation where different people who were both in all parts of the meeting series have different memories. Exactly. And then the analogy to agents – I've been in meetings where something was said and agreed, and then you come back and "but that's what we decided." That's a very common problem. And specifically this – instead of something taking one meeting, it becomes three or four. It's above all the time aspect. You don't have time either – even if the information is important and should be shared, there's a time factor. Conversely, you might feel this is critically important to repeat, but you just don't have time. That's a big problem.

Where Is Memory Heading?

Mitra:
‍But what you were touching on earlier is removing steps from these processes, streamlining them. Where is memory heading?

Martin:I'm really fascinated by how you can vectorize text. You can take a text and create a mass of numbers from it, so that you can measure the semantics – measure what the text means. A trick used to reduce the size of LLMs – which are also essentially just these vectors, long sequences of numbers – is that if you remove the last digits behind the decimal point, you get a less precise AI. You could do the same with a semantic vector – just truncate the last parts of a text. Then you get a less precise representation of what that text means. If you can feed that into an AI instead, you can get concepts, you can get patterns, but you don't get specific details. If you combine that with the traditional way we handle memory today, you can get the superpower we humans have – remembering concepts while being able to forget things – but still recall specific facts with the traditional memory layer. It compresses the memory.

Glenn:
‍That's genuinely fascinating! It's... that's the first time... So you're thinking it's about shortening the text we're vectorizing?

Martin:
‍I was thinking about shortening the vectors themselves, so they become less precise. You still want the vectors and the meaning – you just don't want full precision. Now let it be a little more fuzzy.

Glenn:
‍Absolutely. What I've come across related to indexing is a common technique to fit things into database systems – you simply don't have room for very large vectors.

Martin:
‍Right, it's more of a necessity, but you're thinking it could actually be a feature rather than a compromise.

Glenn:
‍Exactly. And the memory you create with truncated vectors – if we call it that – you could send it in as a pre-context to the agent, before you start sending in specific instructions and text. You could provide this upfront. Then you get an agent that has an abstract way of thinking and continuously updates its understanding.

Wrap-up

Mitra:
‍Time goes very, very quickly. We'll leave you with a short summary from today's episode about memory. What would you like listeners to take away?

Glenn:
‍An agent will never remember everything. Someone has to make sure it gets all the context it needs. Every time I interact with an agent – that's what I think is the most important thing to take away today. They have no memory.

Martin:
‍Sometimes it may be implemented, but as a fundamental principle there is no memory. You have to express everything again and again.

Glenn:
‍Exactly. Yes, I agree. It's very easy to be fooled by your interaction with a chatbot. "I click on you and ask what you know about me – it knows all sorts of things." But that's just because it has things stored in a memory within the conversation we're sitting in. It doesn't necessarily have a memory about me. It can update its memory and has stored things there – I can correct it if it's stored something wrong.

Mitra:
‍This is going to be really interesting to follow. You both touched on how far we haven't gotten yet in terms of using AI fully in our processes. So I'm convinced that as we start putting these tools to use – sitting in work processes and giving it context again and explaining again – we'll discover a whole lot of things that limit and change things, connected specifically to memory beyond all the other smart built-in features. So it's going to be exciting to follow.

Glenn:
‍Thanks so much for this!

Martin:
‍Really great to have this conversation – this time about memory.

Mitra:A big thank you to everyone who listened today. We look forward to hearing from you again. Until next time!

Glenn & Martin:Take care! Bye!

You've been listening to Syntra, a podcast from Fiwe.