Chain-of-thought prompting: a complete guide

Ask a large model a multi-step word problem and it often blurts out a wrong number with total confidence. Chain-of-thought (CoT) prompting fixes that by making the model show its work first — generate the intermediate steps, then the answer. The flip is almost embarrassingly simple, but on GSM8K math problems it took PaLM 540B from 17.9% to 58% accuracy (Wei et al., 2022, NeurIPS).

See it work

Same model, same question. The only change is asking it to reason out loud.

Prompt (standard):
Q: Roger has 5 tennis balls. He buys 2 more cans of balls.
   Each can has 3 balls. How many does he have now?
A: 11

Q: The cafeteria had 23 apples. They used 20 for lunch and
   bought 6 more. How many apples do they have?
A: 27   ← wrong (it pattern-matched "add everything")

Prompt (chain-of-thought):
A: The cafeteria started with 23 apples. They used 20, so
   23 - 20 = 3. They bought 6 more, so 3 + 6 = 9.
   The answer is 9.   ← correct

The model didn't get smarter. It got room to compute. Each step it writes becomes context for the next one, so the final token isn't a wild leap from the problem — it's the last link in a chain it already laid down.

The mental model

Think of a coworker who's brilliant but blurts answers. Ask "what's the total?" and they guess. Ask them to talk through it and they're suddenly reliable — because saying "okay, 23 minus 20 is 3" forces them to actually do the arithmetic instead of vibing toward a plausible number.

Chain-of-thought turns a single hard prediction — problem to answer — into a chain of easy ones, where each step conditions the next.

How it works

1. Encode the problem. The model reads the task plus any worked examples, which prime the reasoning format.
2. Generate steps sequentially. It writes intermediate calculations and logical moves, each one conditioning the tokens that follow. Connectors like "therefore" and "so" structure the flow.
3. Self-correct implicitly. Because every step has to stay coherent with the ones before it, attention can nudge later predictions back on track.
4. Extract the answer. A final step — usually signalled by "the answer is" or "####" — reads off the result conditioned on the whole chain.

Standard prompting models P(answer | problem). CoT models P(answer | problem, step₁, … stepₙ) — a Markov chain of reasoning where accumulated context does the heavy lifting.

Why it works

The size of the gain is dominated by a few factors, roughly in this order:

The deeper reason: predicting "2 + 2 = 4, then 4 × 3 = 12" is an easier sequence of next-token guesses than leaping straight to "12" from "(2+2)×3". Each written step is both output and input, so it doubles as a scaffold and a verification surface.

Where it shines

CoT pays off whenever a task needs two or more reasoning steps you can express in words:

• Math and arithmetic — GSM8K, SVAMP, MultiArith, AQuA-style word problems.
• Commonsense and multi-hop QA — StrategyQA, HotpotQA, anything needing inference across facts.
• Logic and planning — constraint puzzles, scheduling, symbolic deduction.
• Code reasoning — algorithm design, systematic debugging, complexity analysis.

The headline benchmarks from the original work:

Self-consistency — sampling several chains and majority-voting the answer (Wang et al., 2022) — stacks on top: +17.9% on GSM8K, +11.0% on SVAMP, +12.2% on AQuA, +6.4% on StrategyQA.

The same idea, baked into model architecture, is what powers today's reasoning models. OpenAI's o1 scored 74% on AIME 2024 versus GPT-4o's 12%; o3 hit 98.4% on AIME 2025 (o4-mini reached 99.5% with Python tools). Gemini 2.5 Pro reaches 86.7% on AIME 2025 unaided, 24.4% on the ultra-hard MathArena set where rivals score below 5%, and leads the AMO problem set at 25%. Claude 3.7 Sonnet posts 62.3% on SWE-bench and ~80% on AIME in extended-thinking mode.

When to use it (and when not)

Reach for CoT when:

• The problem takes 2+ logical steps and standard prompting gets it wrong.
• You need to see the reasoning — for verification, debugging, or teaching.
• Accuracy gains of 10–40% justify the extra latency.

Skip it when:

• You're on a native reasoning model (o1, o3, Gemini 2.5, Claude 3.7 extended thinking) — external CoT interferes with the built-in process.
• The task is single-step retrieval or pure pattern matching. On some implicit-statistical-learning tasks CoT actively hurts: 94% zero-shot drops to 62.52% with CoT.
• Latency is critical (you need sub-2-second responses) or the model is below 100B parameters and just produces incoherent chains.

When a single chain isn't enough, escalate: to few-shot CoT for domain consistency, to self-consistency when you can spend 5× compute for +10–20% accuracy, to Tree of Thoughts when the problem needs exploration and backtracking, or to a native reasoning model when budget allows maximum quality.

Structure and components

A zero-shot chain needs almost nothing — the problem plus a trigger phrase. Just adding "Let's think step by step" elicits reasoning with no examples at all (Kojima et al., 2022), then (optionally) a second pass extracts the answer:

What is (15 + 27) × 3?

Let's think step by step.

A few-shot chain shows the format through 3–8 worked demonstrations before the real question:

Q: Roger has 5 tennis balls. He buys 2 cans, 3 balls each.
   How many now?
A: Roger started with 5. 2 cans × 3 = 6 new balls.
   5 + 6 = 11. The answer is 11.

Q: [your new problem]
A:

The design principles that make a chain work: sequential connectors ("first," "then," "finally"), explicit calculations ("5 + 3 = 8", never "adding gives 8"), one consistent format across every example, and correct reasoning in the demonstrations — a single wrong example poisons the pattern. For high-complexity problems, add explicit verification steps; for ambiguous ones, add a clarification step that states assumptions up front.

Implementation

The canonical few-shot setup is just a prompt prefix and a normal completion call:

import openai

few_shot = """
Q: Roger has 5 tennis balls. He buys 2 cans, 3 balls each. How many now?
A: Roger started with 5. 2 cans × 3 = 6. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. They used 20 and bought 6 more. How many now?
A: Started with 23. Used 20: 23 - 20 = 3. Bought 6: 3 + 6 = 9. The answer is 9.
"""

def solve_with_cot(problem):
    prompt = f"{few_shot}\nQ: {problem}\nA:"
    resp = openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3,
        max_tokens=300,
    )
    return resp.choices[0].message.content

When one path isn't reliable enough, sample several at higher temperature and vote — this is the core of self-consistency:

from collections import Counter

def self_consistency(problem, n=5):
    answers = []
    for _ in range(n):
        prompt = f"{problem}\n\nLet's think step by step."
        resp = openai.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.8,        # diversity across paths
            max_tokens=500,
        )
        answers.append(extract_answer(resp.choices[0].message.content))
    return Counter(answers).most_common(1)[0][0]   # majority vote

Configuration

On Claude, prefer the native extended-thinking mode over manual prompting. On Gemini, lean on numbered structure. On open-source models, stay at 70B+, use 6–8 explicit examples, and keep temperature low (0.1–0.3).

A typical workflow

Start with zero-shot CoT (five minutes). If reasoning is inconsistent, write 3–5 clear few-shot examples and test on 10 problems. Iterate on failures, then validate on 30–50 held-out problems against a standard-prompting baseline. Only add self-consistency if the accuracy is genuinely critical.

Do: try zero-shot before building examples; show every calculation; verify your examples are correct; keep formatting identical across them; measure against a baseline.

Don't: use CoT on native reasoning models; apply it to one-step problems; let examples contain errors; expect anything coherent below 100B parameters; ignore the latency bill.

Debugging

Testing and proving it

The only honest claim is one you measured. Hold out test problems you never used for prompt development, then compare CoT against a plain baseline:

import numpy as np
from scipy import stats

def ab_test(problems, n=50):
    standard = [evaluate(standard_prompt(p), p) for p in problems[:n]]
    cot = [evaluate(cot_prompt(p), p) for p in problems[:n]]
    _, p_value = stats.ttest_rel(standard, cot)
    print(f"standard: {np.mean(standard):.1%}  cot: {np.mean(cot):.1%}")
    print(f"improvement: {np.mean(cot) - np.mean(standard):.1%}  p={p_value:.4f}")
    return p_value < 0.05

Track accuracy, but also reasoning quality (logical validity, completeness) via human raters, plus latency and token cost so you're honest about the trade. For squeezing tokens, compress filler — "Let's see, we have…" becomes "Given:" — for 20–40% savings at under 5% accuracy cost. Adding explicit verification steps ("checking: does 12 × 3 = 36? yes") buys back 5–15% on math.

Limitations

• Model size is a hard floor. Below ~100B parameters, chains are incoherent and worse than direct answers. This can't be prompted around.
• Cost is inherent. 3–5× tokens and 35–600% more latency come with the territory.
• Faithfulness is unproven. Models may decide the answer first and retrofit plausible reasoning — studies show you can replace meaningful reasoning tokens with nonsense and keep the accuracy. A convincing chain is not a guarantee of genuine reasoning.
• Some tasks degrade. Perception-heavy and implicit-pattern tasks often do worse with CoT; medical/clinical text shows systematic hallucination and omission failures.
• Marginal value is falling. Newer, more capable models gain less, and native reasoning models gain nothing from external CoT.
• Errors cascade. One wrong early step propagates through the chain — longer chains mean more opportunities to compound a mistake.

Ecosystem and what's next

CoT is the trunk of a whole family of reasoning techniques:

Tooling support is broad — LangChain's FewShotPromptTemplate, DSPy's ChainOfThought module with automated example optimization, and LlamaIndex query engines all wrap these patterns. The most productive hybrids pair CoT with retrieval-augmented generation, or RAG (ground each step in retrieved facts to cut hallucination) or with self-consistency plus verification (sample, verify each path, vote among the valid ones).

The transition path in practice: baseline → zero-shot CoT → few-shot → self-consistency → native reasoning models. And the frontier keeps moving — Chain of Draft (2025) trims tokens by "thinking faster by writing less," Auto-CCoT generates contrastive good/bad examples from the model's own errors, and the DUP method (deeply understanding problems) hit 97.1% on GSM8K zero-shot by emphasizing comprehension over chain length. The clear arc is from external prompting trick toward reasoning baked into the architecture.

Summary

• Chain-of-thought makes a model generate intermediate steps before its answer, turning one hard prediction into a chain of easy ones.
• The original lift was dramatic — 17.9% → 58% on GSM8K (PaLM 540B), with self-consistency reaching 74%.
• It only emerges at scale: ~100B+ parameters, with model size accounting for roughly half the effect.
• Use it for multi-step math, commonsense, logic, and code reasoning; skip it for one-step tasks, perception, and native reasoning models.
• It costs 3–5× tokens and 35–600% latency, and its marginal value shrinks on frontier models — always A/B test against a baseline.
• A fluent chain isn't proof of genuine reasoning; verify with self-consistency or symbolic checks before trusting it on high-stakes calls.
• Escalate when a single path stalls: few-shot → self-consistency → Tree of Thoughts → native reasoning models.