Insights a 25 Year Old Movie Can Give Us on LLMs

I often hear complaints that LLM coders are not like junior colleagues. The difference being that junior colleagues learn and grow and don’t make the same mistakes over and over. Time spent attending to a colleague bears long-term fruit.

LLMs that don’t learn from their experiences will be a constant time drain.

25 years ago, the movie Memento explored this very idea.

The main character is Leonard, who has lost the ability to make new memories due to a traumatic injury. His short term memory remains functional, but his long term memory is limited to events prior to the injury.

Leonard has a mission. Someone murdered his wife right at the time of injury. It’s the last thing he remembers. He is hell-bent on finding the killer.

Let’s compare him with an LLM:

• He has a limited context window (his short term memory).
• He has a fair amount of knowledge (his long term memory).
  • The knowledge has a cutoff date (the day of his injury).
• He hallucinates: Gaps in his information don’t stop him. He fills them in.
• He has a system prompt: His goal is to find the killer.

How can Leonard possibly find the murderer with such drawbacks?

He makes notes for each clue. He has documents tying the clues together. He keeps them on his person (and body) for quick access. Much like a simple RAG workflow. Or keeping a handy Markdown file while coding.

.

He has to be disciplined in his notes: The totality of all the evidence needs to be succinct enough for him to digest it all within his short term memory. It needs to fit in his context window.

.

Although the movie doesn’t mention it, I’m sure he has to compact his context from time to time.

There are even hints that he falls prey to prompt injection attacks!

In the future, we may give LLMs more power.

Scratch that.

In the future, we will give LLMs more power.

And they’ll behave like Leonard.

What can we do to prevent Leonard from misbehaving, and protect him from bad actors?