Memory-of-thought (MoT): a complete guide

Every time you call an LLM, it forgets. Each prompt is a stateless event — the model never builds on the reasoning it nailed yesterday. Memory-of-thought (MoT) fixes that without touching a single weight: the model pre-thinks through a pile of unlabeled questions, keeps only the reasoning chains it's confident about, and recalls the most relevant one when a new question shows up. In the original paper, that simple loop lifted ChatGPT by 3.7–9.1% across arithmetic, commonsense, factual, and inference benchmarks (Li & Qiu, EMNLP 2023).

The flip is this: instead of feeding the model fixed human-written examples, you let it write its own examples first, then pick the best-matched one per query. It's self-improvement that's non-parametric — the model gets better without fine-tuning, gradients, or labeled data.

See it work

Take a rate problem. With plain few-shot CoT, the model gets a couple of generic examples that may or may not resemble the question, and it's on its own:

Prompt:
Q: A train travels 60 miles in 1.5 hours. What is its average speed?
A: Let's think step by step.

Output:
60 divided by 1.5... that's about 45 mph. The answer is 45 mph.   ← wrong

With MoT, the model has already pre-thought a memory bank. At test time it retrieves the closest solved problem and reasons by analogy:

Prompt:
Here is a similar solved problem:
Q: A car covers 150 miles in 2.5 hours. What is its average speed?
A: Average speed = distance / time = 150 / 2.5 = 60 mph. The answer is 60 mph.

Now solve:
Q: A train travels 60 miles in 1.5 hours. What is its average speed?
A: Let's think step by step.

Output:
Average speed = distance / time = 60 / 1.5 = 40 mph. The answer is 40 mph.   ← right

Same model, same question. The only change is the in-context example — and it was chosen for this query, not picked once and frozen. (The numbers above are illustrative of the mechanism, not benchmark results.)

The mental model

Think of an expert who never solves a problem from scratch. They recognize "oh, this is like that case from last month" and adapt the approach that worked. Cognitive science calls this case-based reasoning: retrieve a similar past case, adapt its solution.

MoT gives the model a notebook of its own best worked examples, then hands it the right page for each new question.

The student analogy is just as apt. MoT doesn't make the model inherently smarter — it lets the model review relevant worked examples right before the test. The latent capability was always there; self-consistency already showed that correct reasoning paths usually exist among sampled outputs. MoT just externalizes those correct paths and reuses them. And because research shows the choice of in-context examples can swing accuracy by 20%+, picking the right page per question — not a fixed set — is where much of the gain comes from.

How it works

MoT runs in two stages: a one-time pre-thinking pass that builds the memory, and a per-query recalling pass at test time.

Stage 1 — pre-thinking (build the memory, once):

1. Assemble unlabeled data. Collect questions representative of the target task; the paper uses training splits without labels.
2. Sample multiple paths. For each question, generate multiple CoT chains (typically 16) at temperature above 0 so they diverge.
3. Majority vote. Tally the candidate answers across paths; the most-voted answer wins.
4. Filter for confidence. Keep only questions where consensus is strong enough (e.g., 12 of 16 paths agree).
5. Pick a representative chain. Randomly select one chain that agrees with the majority answer.
6. Store it. The (question, reasoning chain, answer) tuple joins the memory bank.

Stage 2 — recalling (per test question):

1. Embed the test question. The paper uses the LLM's own representations for similarity.
2. Retrieve. Find the stored question most semantically similar to the test question.
3. Construct the prompt. Place the retrieved memory as an in-context demonstration, then the test question.
4. Generate. The model reasons with the retrieved chain as scaffolding.

Inference is single-pass — one retrieval, one generation. Only pre-thinking is iterative and multi-sample, and it's amortized across every future query.

Why it works

Four causal pathways carry the gains: relevant priming (a closely related solved problem activates the right strategy), noise reduction (majority voting filters out flawed chains before they can be copied), structural-similarity transfer (the chain becomes a template to adapt), and confidence calibration (high-consensus questions mark genuine competence). Not all factors matter equally:

The dependency is genuine, not cosmetic. Ablations show that removing pre-thinking (random examples instead of self-generated ones) or removing recalling (memories built but never retrieved) both collapse performance to near baseline. Neither half works alone.

Where it shines

MoT helps wherever reasoning patterns recur, so a solved neighbor transfers cleanly. The paper measured consistent gains on ChatGPT (GPT-3.5-Turbo) across four task families:

• Arithmetic (AQuA) is the strongest case — set-up-equation, substitute, solve transfers directly between analogous word problems.
• Commonsense (OBQA, com_v, BoolQ) reuses world-knowledge reasoning patterns across structurally similar scenarios.
• Factual (DROP, fact_checker, qa_wikidata) benefits from stored "find fact → apply rule → derive conclusion" templates.
• Natural language inference (ANLI A1/A2/A3) reuses consistent entailment/contradiction logic.

The gains are consistent rather than dramatic — characteristic of a method that refines existing capability instead of unlocking new behavior. Crucially, MoT improves performance regardless of which CoT variant you use as the base (standard, complex, zero-shot CoT), which means it addresses a different bottleneck than the reasoning format itself. Beyond the paper, the same pattern fits education (worked-example tutoring), code generation (the later Modularization-of-Thought work, 2025, explored this), medical and legal reasoning (both lean heavily on precedent/case matching), and scientific reasoning.

When to use it (and when not)

Reach for MoT when:

• Few-shot CoT is moderate but not satisfying, and there's headroom to improve.
• Unlabeled domain questions are plentiful but annotations are expensive.
• The task benefits from analogical reasoning — similar problems share solution structure.
• You can pay a one-time pre-computation cost and need gains without retraining.
• Different questions need different reasoning, so dynamic example selection pays off.

Skip MoT when:

• Zero-shot already nails it (no headroom).
• No unlabeled domain data exists.
• The task changes fast and memories go stale (e.g., current-events QA).
• Latency rules out retrieval, or the domain is so novel the model can't produce useful chains even with multiple samples.

Model fit: any model with solid in-context learning and CoT works. Practically that means ~7B+ parameters for open models; below ~1B the emergent reasoning isn't there. GPT-3.5-Turbo is the proven sweet spot — strong enough to generate useful memories, weak enough to have headroom. GPT-4-class models produce higher-quality memories but show smaller relative gains. Open models like Llama want 70B+ for the pre-thinking pass, though smaller 7B–13B models work fine as inference models with memories generated by a larger one — cross-model transfer (pre-think with GPT-4, infer with Llama-7B) is a viable variant. Claude (Anthropic) brings a 200K context that lets you retrieve more memories at once. Models with built-in reasoning (o1, o3, Gemini 2.5) may benefit less, since they already internalize multi-step reasoning.

When to escalate: if MoT's 3–9% isn't enough, consider fine-tuning on labeled data. If pre-thinking keeps failing to produce high-confidence memories, upgrade the base model. If retrieval keeps returning junk, the domain may need a specialized similarity metric. If the real bottleneck is knowledge rather than reasoning, combine MoT with RAG.

Alternatives worth considering directly: standard few-shot CoT when you already have great hand-crafted examples; self-consistency when you want gains with no pre-computation; Auto-CoT when you want automated example selection without a persistent bank; RAG when the bottleneck is knowledge, not reasoning; fine-tuning when you have labels and need bigger jumps than MoT delivers.

Structure and components

MoT has four parts — the first three are required, the fourth is technically optional but practically essential:

1. Unlabeled dataset — questions representative of the target task; no labels needed. Cover the diversity of expected test queries.
2. Memory bank — the indexed store of (question, reasoning chain) pairs from pre-thinking. Typically 500–5,000 entries; even a tight 50–100 entries can beat generic few-shot.
3. Retrieval mechanism — matches test questions to memories. The paper uses the LLM's own embeddings; alternatives include sentence transformers, BM25, or hybrids.
4. Confidence filter — the majority-voting gate. Skip it and unfiltered noise degrades performance.

The recall-time prompt mirrors few-shot CoT but with dynamically selected content. The principle is structural parallelism — the retrieved example must follow the format you want back:

Q: [Retrieved similar question from memory]
A: Let's think step by step.
[Retrieved reasoning chain]
The answer is [retrieved answer].

Q: [Test question]
A: Let's think step by step.

Design guidelines that matter: store clear, complete chains (truncated ones hurt); favor a smaller well-curated bank over a large noisy one; keep the stored format identical to the expected output format; and draw the unlabeled dataset from the same domain as the target task — cross-domain retrieval typically harms performance.

For ambiguous tasks, retrieve top-k instead of top-1 for diverse perspectives. For complex multi-step reasoning, use a CoT base that produces detailed chains, store longer chains, and consider running MoT per sub-problem. For format-critical tasks, add explicit format instructions and store format-correct examples. For low-resource settings, generate synthetic question variations to expand the pre-thinking set.

Implementation

Core mechanism: pre-think and filter

The heart of MoT is generating paths, voting, and keeping only confident chains:

import random
from collections import Counter

def majority_vote(paths, extract_answer):
    answers = [extract_answer(p) for p in paths]
    majority_answer, count = Counter(answers).most_common(1)[0]
    return majority_answer, count / len(answers)  # answer, confidence

def build_memory_bank(questions, generate_paths, extract_answer,
                      confidence_threshold=0.75):
    memory_bank = []
    for q in questions:
        paths = generate_paths(q, num_paths=16, temperature=0.7)
        answer, confidence = majority_vote(paths, extract_answer)
        if confidence >= confidence_threshold:
            agreeing = [p for p in paths if extract_answer(p) == answer]
            memory_bank.append({
                "question": q,
                "reasoning": random.choice(agreeing),
                "answer": answer,
                "confidence": confidence,
            })
    return memory_bank

Retrieval

Index the bank by question embedding and pull the nearest neighbor(s):

from sentence_transformers import SentenceTransformer
import numpy as np

class MemoryRetriever:
    def __init__(self, memory_bank):
        self.memory_bank = memory_bank
        self.encoder = SentenceTransformer("all-MiniLM-L6-v2")
        self.embeddings = self.encoder.encode([m["question"] for m in memory_bank])

    def retrieve(self, test_question, top_k=1):
        q = self.encoder.encode([test_question])
        sims = np.dot(self.embeddings, q.T).squeeze()
        idx = np.argsort(sims)[-top_k:][::-1]
        return [self.memory_bank[i] for i in idx]

Test-time inference

Build the prompt from the retrieved memory and generate deterministically:

def mot_inference(test_question, retriever, client, model="gpt-3.5-turbo"):
    memory = retriever.retrieve(test_question, top_k=1)[0]
    prompt = (
        f"Here is a similar solved problem:\n"
        f"Question: {memory['question']}\n"
        f"Solution: {memory['reasoning']}\n"
        f"Answer: {memory['answer']}\n\n"
        f"Now solve the following problem:\n"
        f"Question: {test_question}\nSolution:"
    )
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0, max_tokens=512,
    )
    return resp.choices[0].message.content

The original implementation uses GPT-3.5-Turbo with parallel API calls (it supports multiple API keys for throughput). Adapt to Claude via the Messages API; its long context allows more memories. LangChain offers ConversationBufferMemory and VectorStoreRetrieverMemory as building blocks; DSPy maps MoT to its retrieve-then-generate modules; LlamaIndex's vector store can back the memory bank. You'll need API access to a capable LLM, Python 3.8+, an embedding model, and storage (JSON file, vector DB, or in-memory).

Configuration

Tune by task: arithmetic uses the paper defaults (16 paths, 0.75 threshold, 0.7 temp); commonsense may want a lower 0.6 threshold given answer diversity; NLI/classification need only 8–16 paths. For domain work, build a domain-specific bank, swap in a domain embedding model (SciBERT for science, LegalBERT for law, BioBERT for biomedical), and lower the threshold for harder domains so enough memories survive.

Workflow

Define the task and collect unlabeled questions (500–2,000 is typical) → run pre-thinking → filter and store → spot-check stored memories for correctness → set up embedding retrieval → evaluate against baseline CoT on a held-out set → tune threshold, top_k, and prompt format → deploy → monitor and periodically refresh.

Do: use unlabeled data from the same distribution as the task; validate memory quality before deploy; tune the threshold on a validation set; use the same LLM for pre-thinking and inference when you can; log which memories get retrieved and whether they help.

Don't: store chains from a weaker model for use by a stronger one (it can constrain the stronger model); skip the confidence filter; let the bank grow huge without efficient indexing; assume cross-domain transfer; or retrieve so many memories that context overflow and distraction outweigh the diversity.

Debugging

• Low overall improvement → bank quality is poor (raise num_paths, raise confidence_threshold, check the dataset is representative), or retrieval returns junk (better embedding model, hybrid retrieval), or the base model is too weak (upgrade, or pre-think with a stronger model and infer with a cheaper one).
• Inconsistent outputs on similar queries → different memories retrieved for near-identical questions; normalize retrieval or raise top_k and aggregate.
• Worse than baseline on some questions → a misleading memory; add a similarity threshold and fall back to plain CoT when the best match is below it. If the stored chain itself is wrong despite voting, raise the threshold.
• Format violations → ensure consistent formatting across memories and add explicit format instructions.
• Hallucinated reasoning → the model is over-copying; add "use the example as guidance, but reason independently."

Common mistakes: using labeled data for pre-thinking (defeats the purpose), inconsistent answer extraction during voting (false vote splits), too-low pre-thinking temperature (no diversity), never refreshing as the distribution drifts, and applying MoT where the model is already near-perfect.

Testing and optimization

Reserve ~20% of any labeled data as a holdout and compare MoT against zero-shot, few-shot, and baseline CoT; use k-fold cross-validation when labels are scarce. Because the headline gains are a modest 3–9%, statistical rigor matters: use paired tests (McNemar's for classification, paired t-test for continuous metrics) and bootstrap or permutation confidence intervals so you don't mistake noise for improvement. Track accuracy/F1 and exact match alongside MoT-specific signals — retrieval precision (fraction of relevant retrievals), confidence calibration, improvement-over-baseline delta, and run-to-run consistency. Set temperature=0 for inference and fix seeds during pre-thinking for reproducibility.

Optimize by storing condensed chains (drop the "Let's think step by step" preamble, keep the logical steps), caching embeddings, pruning never-retrieved memories, and weighting retrieval by stored confidence. For scale, use approximate nearest-neighbor search (FAISS, Annoy) and batch test questions by their nearest memory.

Limitations

Some ceilings are fundamental. MoT can't exceed the base model — if the model can't solve a class of problems, the bank ends up empty or filled with confident-but-wrong chains. Semantic similarity isn't reasoning similarity: "how many ways to arrange 5 books?" and "how many ways to choose 3 of 5 books?" look alike but need different reasoning, and surface-similar retrieval can mislead. The memory is static after pre-thinking — new problem types need a fresh pass. And there's no self-correction within single-pass MoT: a bad retrieval just produces confident, well-structured, wrong reasoning.

It also handles some situations poorly: highly diverse domains need very large (expensive) banks; rapidly evolving tasks erode the one-time-cost advantage through constant refresh; and multi-modal reasoning falls outside a text-only bank. Edge cases — ambiguous inputs, conflicting constraints (memory maximizes, question minimizes), out-of-domain questions that retrieve the "least dissimilar" junk — all push performance toward or below baseline.

For token/context pressure, summarize long chains before storage, retrieve fewer memories, and place them at the start of the prompt. For robustness, add retry-with-backoff during pre-thinking, fall back to random memory selection on retrieval failure, and use robust answer parsing during voting.

Advanced techniques

Once the basics work, several extensions help. Hierarchical / decomposed memory: break complex questions into sub-problems and retrieve a memory per sub-problem, or store separate entries for "set up the equation" vs. "solve the equation," or do layered retrieval (strategy memory first, then a detail memory). Self-verification: after MoT answers, run a verification prompt, or compare against a zero-shot answer — disagreement flags a likely retrieval error. Confidence-weighted and adaptive retrieval: weight memories by stored confidence, and retrieve longer chains for hard questions, shorter for easy ones. Style control: the retrieved memory implicitly sets reasoning style, so pre-filter the bank for the tone you want.

Risk and ethics

MoT's big reveal is that LLMs are often more capable than their zero-shot performance suggests — simply showing the model its own prior correct reasoning closes a meaningful gap, implying the barrier is frequently contextual framing, not raw capability.

On the upside, MoT is comparatively auditable — improvements trace to specific memory entries, unlike fine-tuning. Two safety notes: if the unlabeled dataset is sourced externally, validate it against prompt injection before chains enter the bank; and watch for over-reliance, since structured-looking reasoning reads as more authoritative than it may be.

Ecosystem and integration

The seminal paper — "MoT: Memory-of-Thought Enables ChatGPT to Self-Improve" by Xiaonan Li and Xipeng Qiu (Fudan University), EMNLP 2023, pages 6354–6374, tested on ChatGPT (GPT-3.5-Turbo-0301). The official implementation is on GitHub (LeeSureman/MoT) with parallel API calls, multi-account support, and the full pre-thinking → filtering → recalling pipeline. It ships pre-computed memory banks and evaluation scripts for AQuA, DROP, ANLI (A1/A2/A3), OBQA, com_v, BoolQ, fact_checker, and qa_wikidata — so you can skip pre-thinking on supported datasets. For production, back the bank with a vector DB (Pinecone, Weaviate, Milvus), version it alongside the model, and track experiments with LangSmith or Weights & Biases.

MoT didn't appear from nowhere. It builds on chain-of-thought (Wei et al., 2022), borrows majority voting from self-consistency (Wang et al., 2022), echoes "LLMs can self-improve" (Huang et al., 2022) but stays non-parametric, and adapts the retrieval idea from kNN-Prompting (Xu et al., 2023) — swapping labeled examples for self-generated chains. The original v1 (May 2023) was titled "MoT: Pre-thinking and Recalling Enable ChatGPT to Self-Improve with Memory-of-Thoughts"; the v2 EMNLP version (October 2023) refined the framing. It has since influenced Think-in-Memory (TiM), Buffer of Thoughts (2024), and continuous-memory work like "From RAG to Memory" (2025).

How does MoT stack up against the obvious alternatives?

The two strongest hybrids: MoT + self-consistency (MoT-SC) stacks better examples (MoT) with better answer selection (voting at inference), and MoT + RAG addresses orthogonal bottlenecks — RAG supplies knowledge, MoT supplies reasoning patterns. Other combinations pair MoT with Tree-of-Thoughts (memories as starting nodes) or Reflexion (self-correcting the MoT output). In agent frameworks (ReAct, AutoGPT), the bank becomes a "playbook" of proven strategies the agent retrieves before acting. To migrate from plain CoT: keep your CoT prompts as the base method, collect unlabeled questions, run pre-thinking with those prompts, build the bank, and deploy only if it beats baseline on a validation set.

Future directions

The frontier is moving from a one-time pass toward continuous, real-time memory that learns from feedback without ever updating weights. Other open threads: cross-modal memory (chains over images, code, structured data), personalized or federated banks (per-user, per-org, or shared-but-private collective improvement), memory distillation (using quality-filtered chains as fine-tuning data), and smarter retrieval that goes beyond embedding similarity toward structural or reasoning-type matching. Open questions remain around optimal bank composition, memory scaling laws (when do returns diminish or turn negative?), cross-task transfer, and how MoT interacts with built-in-reasoning models like o1, o3, and Gemini 2.5.

Summary

• MoT is non-parametric self-improvement: the model pre-thinks on unlabeled data, keeps only high-confidence reasoning chains, and recalls the most relevant one per query — no fine-tuning, no labels.
• Two stages: a one-time pre-thinking pass (sample ~16 paths, majority-vote, filter, store) and a per-query recalling pass (embed, retrieve, prompt, generate). Both are necessary — ablations show neither alone works.
• Modest but consistent gains: 3.7–9.1% on ChatGPT across arithmetic, commonsense, factual, and NLI, with AQuA going 49.7% → 54.1% (Li & Qiu, EMNLP 2023).
• It layers on any CoT variant and addresses a different bottleneck than the reasoning format itself.
• Cost is front-loaded: ~$5–30 of one-time pre-thinking for 1,000 questions, then a negligible per-query bump.
• Reach for it when few-shot CoT is mediocre, unlabeled data is cheap, and reasoning patterns recur; skip it when zero-shot suffices, data is unavailable, the task drifts fast, or the base model can't reason at all.
• Watch the silent failure: an irrelevant memory yields confident-looking wrong reasoning — guard it with a retrieval similarity floor and a plain-CoT fallback.