I've been thinking about what the next generation of AI might look like.
Not necessarily bigger models, more parameters, larger context windows, or more training data. Those things can certainly improve capability, but I wonder whether there are more fundamental differences between today's AI systems and human intelligence that we should be exploring.
One thing that stands out to me is how humans learn, adapt, question their own thinking, recognise mistakes, and gradually improve the way they approach problems.
Could we build AI systems that do more of these things?
I don't mean simply giving an LLM access to more information or allowing it to remember previous conversations. I'm thinking about something deeper: an AI system that can learn while it works, evaluate its own reasoning, recognise weaknesses in its approach, and potentially improve the mechanisms it uses to learn and reason.
This is a thought-sharing exercise rather than a prediction. Some of the ideas below are already active areas of research, while others are questions about how those ideas might fit together.
1. The difference between training and learning
Most traditional large language models have a fairly clear separation between training and inference.
During training, the model adjusts its weights based on examples and feedback. This is where the model learns its statistical representations and capabilities.
During ordinary inference, however, those weights remain fixed. The model receives a prompt, processes it using what it has already learned, and generates a response.
It can use the current conversation as context, but that doesn't normally mean its underlying parameters have changed or that it has permanently learned something new.
Humans work differently.
We don't stop learning when we leave school, finish a course, or complete a particular task. We continually acquire information, adapt our behaviour, revise assumptions, and change how we approach problems.
Our brains are constantly processing experiences. Learning can happen through deliberate study, but also through observation, experimentation, feedback, and reflection.
Neuroplasticity allows the brain to change its connections and organisation in response to experience.
This raises an interesting question:
Could an AI system learn continuously while it is being used, rather than relying primarily on separate training and inference phases?
There is already research moving in this direction. Continual learning, online learning, and test-time training all explore ways for models to adapt beyond conventional offline training.
For example, Google's Titans: Learning to Memorize at Test Time explores a neural long-term memory module that can learn to retain information during test-time processing.
That isn't the same as a human brain continuously rewiring itself, and it doesn't mean a model can freely rewrite all its own capabilities. But it demonstrates that the boundary between training and inference doesn't have to be as rigid as it often is in conventional LLM architectures.
2. Learning without forgetting
If we want AI systems to learn continuously, another problem immediately appears: forgetting.
Humans can learn new skills without necessarily losing everything they learned before. We aren't perfect at this, of course, but we have mechanisms that allow us to retain important knowledge while adapting to new experiences.
AI systems that continually update their parameters can suffer from a problem known as catastrophic forgetting.
A model learns something new, but in the process its performance on previously learned tasks deteriorates.
This is one of the central challenges in continual learning.
There are several approaches to mitigating it, including:
- Experience replay: revisiting examples from previous learning.
- Elastic Weight Consolidation: protecting parameters that are important to previously learned tasks.
- Progressive networks: adding new capacity while retaining older representations.
- Continual backpropagation: approaches designed to help maintain the network's capacity to learn over time.
These approaches are not equivalent to human memory, but they demonstrate that continual learning is an established research problem.
I think any AI architecture that aims to learn continuously would need to address this from the beginning.
It would need some way to decide what information is worth retaining, what can be discarded, and how new learning should affect existing knowledge.
That leads to another question: how does the system decide whether what it has just learned is actually correct?
3. Metacognition: thinking about thinking
One of the most interesting aspects of human intelligence is metacognition.
Metacognition is essentially the ability to think about and evaluate our own thinking.
We don't just solve problems. We can also reflect on how we are solving them.
For example, when working through a difficult problem, a person might recognise:
- I don't fully understand this.
- My current approach isn't working.
- I'm making an assumption that I haven't verified.
- There might be another way to approach this.
- I should investigate this further before reaching a conclusion.
This is more than simply knowing information. It involves monitoring our own reasoning and recognising its limitations.
An experienced engineer, scientist, or mathematician often develops a strong awareness of what they know and, perhaps more importantly, what they don't know.
Expertise doesn't necessarily mean becoming more confident about everything. Sometimes it means becoming better at recognising uncertainty.
Could AI systems develop something similar?
Current LLMs can be prompted to review their answers, identify possible errors, explain uncertainty, and reconsider their reasoning.
There is research into self-evaluation, self-refinement, process reward models, and other techniques that attempt to improve this behaviour.
However, there is an important distinction between producing a statement such as "I might be wrong" and genuinely having a reliable mechanism for recognising when and why an answer is likely to be wrong.
Saying you are uncertain is not the same as being well calibrated.
What interests me is whether metacognition could become a more fundamental part of an AI architecture rather than something we try to encourage through prompting.
4. What if an AI had another AI to challenge it?
One possible approach might be to use two separate LLM instances with different responsibilities.
The first instance would focus on solving a problem, generating ideas, and developing a response.
The second instance would act as a critic or metacognitive evaluator.
Its job would not simply be to produce another answer. It would examine the first instance's reasoning, challenge assumptions, identify gaps, and look for possible mistakes.
The process might look something like this:
- The first model receives a problem and develops a solution.
- The second model reviews the solution and identifies weaknesses.
- The first model considers the criticism and revises its reasoning.
- The second model evaluates the revised solution.
- The process continues until the result meets an agreed standard, or until further revision is no longer useful.
The two instances could also have different roles or objectives.
For example, one might prioritise creativity and exploration, while the other prioritises accuracy, consistency, and evidence.
This is not an entirely new idea.
Research into multi-agent debate has explored multiple language-model instances proposing answers, critiquing one another, and debating over several rounds. A notable example is Improving Factuality and Reasoning in Language Models through Multiagent Debate, published at ICML 2024.
There are also conceptual connections to Marvin Minsky's Society of Mind, which explores the idea that intelligence can emerge from the interaction of many simpler processes.
What interests me is taking this beyond improving a single answer.
Could the critic help the system improve the way the problem-solving model approaches future problems?
That would be a more significant step.
Instead of merely correcting an answer, the system could identify a recurring weakness in its reasoning, investigate why it happens, and adapt its future strategy.
For example, it might notice that it repeatedly makes unsupported assumptions when working with incomplete information.
Rather than simply correcting each individual mistake, it could modify its problem-solving process to explicitly test assumptions before relying on them.
This starts to look less like answer correction and more like learning how to reason better.
5. The importance of recognising your own mistakes
There is another aspect of human intelligence that I think is particularly important: the ability to recognise mistakes without always needing someone else to point them out.
Humans can sometimes notice that something doesn't make sense, even when they don't immediately know the correct answer.
We might revisit a calculation, question an assumption, or realise that a conclusion conflicts with something we already know.
This ability is closely related to metacognition, but it also involves curiosity and the motivation to investigate.
An AI system could potentially use a similar process.
Imagine that a model produces a solution and its evaluator identifies a possible contradiction.
Rather than immediately generating another answer, the system could:
- Identify exactly which part of its reasoning is questionable.
- Determine what information would help resolve the uncertainty.
- Search for evidence or run an experiment.
- Compare the new evidence against its original assumptions.
- Record the outcome so the same mistake is less likely to happen again.
This last step is important.
If a system repeatedly makes the same mistake, corrects it, and then makes it again in a similar situation, has it really learned?
A genuinely adaptive system should ideally be able to retain lessons from its experiences and use them to improve future performance.
Of course, it would need to distinguish between a genuine mistake and a reasonable decision that happened to produce an unexpected result.
It would also need external evidence wherever possible. A model judging its own answer is not necessarily a reliable way of establishing whether that answer is true.
6. Have these ideas already been explored?
Yes. Many of the individual components have substantial research behind them.
A few examples illustrate how much of this territory is already being investigated:
Continual learning and test-time adaptation
Researchers are exploring ways for models to adapt after their initial training, retain information over time, and learn from new experiences. Google's Titans architecture is one example of work on test-time memory.
Self-evaluation and self-refinement
Research explores how models can assess their own outputs, critique their reasoning, and use feedback to improve subsequent attempts. Examples include Self-Taught Reasoner (STaR) and Self-Rewarding Language Models.
Multi-agent debate
Researchers have investigated whether multiple model instances can improve reasoning by proposing, challenging, and refining answers together. The ICML 2024 multi-agent debate paper is a useful example.
Meta-learning
Meta-learning is often described as learning to learn. Rather than only learning a particular task, a system learns aspects of the process that help it learn new tasks more effectively.
Recursive self-improvement
This explores whether an AI system can contribute to improving its own capabilities, including its algorithms, training processes, tools, or architecture.
These are not all the same problem, and they don't necessarily use the same techniques.
But they point towards a common theme: moving beyond systems that simply apply what they learned during training, towards systems that can participate in their own ongoing improvement.
The research is already there in pieces.
The question I find interesting is what happens when we start thinking about these capabilities as parts of one connected architecture.
7. Bringing the pieces together
Imagine an AI system built around several interacting capabilities:
A problem-solving component
Responsible for reasoning, generating ideas, exploring possibilities, and completing tasks.
A metacognitive component
Monitors the problem-solving process, identifies uncertainty, challenges assumptions, and looks for weaknesses.
A memory system
Retains useful experiences, previous mistakes, successful strategies, and information that may be relevant to future tasks.
A learning mechanism
Uses feedback and experience to adapt the system's knowledge, behaviour, or problem-solving strategies.
A verification mechanism
Checks claims against external evidence, tools, experiments, tests, or formal methods wherever possible.
These components could work together in a continuous feedback loop:
- The system attempts a task.
- It evaluates its own approach and result.
- It identifies weaknesses or uncertainty.
- It investigates and verifies the problem.
- It updates its memory or learning strategy.
- It applies what it has learned to future tasks.
Over time, the system could become better not just at answering questions, but at recognising which approaches work, which don't, and why.
There are obvious challenges.
Two models can share the same biases or blind spots. A critic can confidently endorse an incorrect answer. Multiple agents can reinforce one another's mistakes rather than expose them.
This is why independent verification matters.
Where possible, the system should use external sources of truth: executable tests, mathematical proofs, real-world observations, reliable datasets, or other checks that do not depend entirely on another language model agreeing with it.
And there is another distinction worth making: evaluating an answer is not the same as improving the underlying model. A system might become better at selecting, prompting, or organising its existing capabilities without actually changing the knowledge or reasoning mechanisms that produced them.
That distinction becomes especially important when we consider the next idea.
8. The chicken-and-egg problem of intelligence
This is where the idea becomes particularly interesting to me.
What if human-level intelligence requires more than the ability to learn, reason, and evaluate information?
What if it also requires the ability to improve the mechanisms used for learning and reasoning?
Consider the progression:
- First, a system learns information.
- Then, it learns to evaluate its own knowledge and reasoning.
- Then, it learns which reasoning strategies are effective.
- Then, it improves the way it learns and develops those strategies.
- Eventually, it might improve the mechanisms responsible for its own improvement.
This introduces a potential chicken-and-egg problem.
To improve its reasoning, a system needs enough intelligence to recognise the weaknesses in its current reasoning.
To improve its learning mechanism, it needs enough understanding to evaluate how effectively it learns.
To improve its architecture, it may need capabilities that its existing architecture does not yet provide.
How does a system become capable of improving the very mechanisms that determine its current capabilities?
This is closely related to the idea of recursive self-improvement.
In its strongest form, the idea is that an AI system becomes capable of improving itself, and that an improved version becomes more capable of improving itself again.
In theory, this could create a feedback loop in which each generation becomes increasingly effective at developing the next.
But I think it is important to distinguish several levels of improvement.
Improving knowledge means learning new information.
Improving performance means becoming better at completing a particular task.
Improving reasoning means developing better strategies for solving problems.
Improving learning means becoming more effective at acquiring and retaining new capabilities.
Improving the improver means changing the mechanisms responsible for learning, reasoning, and evaluating future improvements.
The last two are particularly interesting because they move beyond simply getting better at a task.
They involve improving the process through which improvement happens.
This is where the idea starts to become recursive.
9. Opening Pandora's box?
If an AI system can improve its own learning mechanisms, reasoning strategies, and potentially its architecture, where does that process stop?
Could it eventually develop approaches that its original designers never anticipated?
Could it discover better ways of representing information, organising memory, allocating computational resources, or solving problems?
Could it design a more capable version of itself?
These are fascinating questions, but they also introduce significant risks.
Self-modification is not automatically beneficial.
A system could change something that appears to improve one benchmark while damaging other capabilities. It could forget previously learned information, introduce instability, or optimise for a measurement that does not reflect genuine improvement.
There is also the problem of evaluation.
If a system is responsible for improving itself, how do we know that the changes are actually improvements?
A model might develop a new reasoning strategy that produces more convincing answers without making those answers more accurate.
It might also become better at evaluating itself according to its own criteria while becoming worse at satisfying the original objectives.
This suggests that any serious self-improvement architecture would need safeguards:
- Independent evaluation of proposed changes.
- Reliable tests and benchmarks that cannot easily be gamed.
- Mechanisms for preserving existing capabilities.
- The ability to roll back harmful changes.
- Constraints on which parts of the system can modify themselves.
- Human oversight for changes with significant consequences.
And we should be careful not to assume that recursive self-improvement automatically leads to an intelligence explosion.
Progress would still be constrained by compute, hardware, data, the difficulty of validating improvements, and the complexity of the problems being solved.
Nor is it established that human-level intelligence requires unrestricted recursive self-modification. That is a hypothesis, not a scientific conclusion.
However, I do wonder whether robust general intelligence will eventually require systems that can improve not just what they know, but how they learn, reason, and adapt.
10. Rethinking what intelligence means
Perhaps one of the limitations of how we currently think about AI is that we focus heavily on the capabilities a model demonstrates at a particular point in time.
How well does it answer questions? How well does it write code? How well does it solve mathematical problems? How much information can it retain?
These are useful measurements, but they mostly tell us what a system can do now.
Human intelligence is also about how we change over time.
We learn from experience. We develop expertise. We recognise our limitations. We question our assumptions. We change our strategies when they stop working.
And, perhaps most importantly, we can sometimes learn how to learn more effectively.
Maybe the next major step in AI isn't simply building a model that knows more or reasons faster.
Maybe it is building a system that can participate in its own intellectual development.
A system that doesn't just produce an answer, but asks whether its approach was appropriate.
A system that doesn't just correct a mistake, but tries to understand why it made it.
A system that doesn't just learn new information, but improves how it acquires, evaluates, and retains that information.
And perhaps, eventually, a system that can safely improve the mechanisms responsible for all of those things.
I don't know whether this is the direction AI development will take, or whether it is even necessary to reach human-level intelligence.
But I think it is an interesting way to look at the problem.
Instead of asking only how we make AI more intelligent, perhaps we should also be asking how we make AI better at becoming more intelligent.
Further reading
Some of the research and ideas that connect to this discussion:
- Titans: Learning to Memorize at Test Time — NeurIPS 2025 — neural long-term memory and learning at test time.
- Improving Factuality and Reasoning in Language Models through Multiagent Debate — ICML 2024 — multiple model instances proposing, critiquing, and debating answers.
- Self-Taught Reasoner (STaR) — bootstrapping reasoning through generated rationales and iterative training.
- Self-Rewarding Language Models — models using their own judgements as part of an iterative improvement process.
- Society of Mind — Marvin Minsky — the idea that intelligence can emerge from interacting processes.
- Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026) — a survey of recent research into different forms of AI self-improvement and their limitations.
These works do not describe one unified solution to the ideas in this article. They are starting points for exploring the individual pieces.
I'd be interested to hear what others think. Is the ability to improve its own learning and reasoning mechanisms a necessary step towards more general intelligence, or are we underestimating what can be achieved with existing architectures?

