Zero-shot chain-of-thought: a complete guide

Ask a large model a multi-step math question and it often blurts out a wrong number. The fix turned out to be almost insultingly simple: append "Let's think step by step" and the model lays out its reasoning before answering. Kojima et al. (2022) showed this one phrase lifts text-davinci-002 from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K — no examples, no fine-tuning, just a trigger.

See it work

Same model, same question. The only change is one line at the end.

Prompt (plain zero-shot):
Q: A juggler has 16 balls. Half are golf balls, and half of the golf
balls are blue. How many blue golf balls are there?
A:

Output:
4

(Wrong. The model guessed in one leap. 16 → half golf → "4" skips a step.)

Now add the trigger:

Prompt (zero-shot CoT):
Q: A juggler has 16 balls. Half are golf balls, and half of the golf
balls are blue. How many blue golf balls are there?
A: Let's think step by step.

Output:
There are 16 balls in total.
Half of them are golf balls, so there are 16 / 2 = 8 golf balls.
Half of the golf balls are blue, so there are 8 / 2 = 4 blue golf balls.
The answer is 4.

Here the answer is the same, but watch what changed: the model wrote down the intermediate quantities (8 golf balls) instead of jumping to a guess. On harder problems that scratch work is the difference between right and wrong — the model commits to each sub-result and builds on it.

The mental model

Picture a colleague who's brilliant but blurts the first thing that comes to mind. Ask "what's the answer?" and they fire off an instinct. Ask them to "walk me through it" and they slow down, show their work, and catch their own mistakes. Zero-shot CoT is that single nudge.

Few-shot CoT teaches reasoning by showing worked examples. Zero-shot CoT just asks for it — and the model already knew how.

How it works

The technique is mechanically a two-stage prompt. First you elicit the reasoning, then you extract the final answer from it. That second stage matters: left to ramble, the model produces a paragraph of reasoning but no cleanly parseable answer.

1. Build the reasoning prompt. Take the question and append the trigger phrase, classically Let's think step by step.
2. Generate the chain. The model produces step-by-step working — the intermediate quantities, the sub-deductions, the logic.
3. Append an extraction cue. Feed the question, the generated reasoning, and a cue like Therefore, the answer (arabic numerals) is back to the model.
4. Read the final answer. The model now emits a short, parseable answer conditioned on its own reasoning.

You can collapse this into one call for casual use, but the paper's reported gains come from the explicit two-stage setup, because it separates "reason freely" from "commit to a parseable answer."

Why it works

The trigger shifts the model from System-1 pattern-matching toward System-2 deliberation. A few factors carry most of the effect:

Where it shines

The headline wins are on multi-step reasoning where a single-shot guess reliably fails. From Kojima et al. (2022), all with text-davinci-002:

• Arithmetic word problems — MultiArith jumped from 17.7% to 78.7%; GSM8K from 10.4% to 40.7%. The technique was also tested on AQUA-RAT, SVAMP, SingleEq, and AddSub.
• Symbolic reasoning — Last Letter Concatenation and Coin Flip, tasks that demand tracking state across steps.
• Commonsense and logical reasoning — Date Understanding and Tracking Shuffled Objects benefited; pure commonsense lookup (CommonsenseQA-style) gained less, since those need knowledge more than steps.

The pattern: the more a task decomposes into sequential sub-results, the bigger the lift. Single-fact recall or sentiment classification barely moves.

When to use it (and when not)

Reach for it when:

• The task needs multiple steps (math, logic, planning, multi-hop questions).
• You have no labeled examples handy, or examples are expensive to write.
• You're prototyping and want a one-line accuracy bump before investing in few-shot.

Skip it when:

• The task is single-step: classification, extraction, lookup, short factual answers. CoT adds tokens and latency for no gain, and can even hurt by overthinking.
• You need terse, low-latency output — the reasoning chain costs tokens both ways.
• You're on a small model that lacks the emergent reasoning skill.

Model fit. Best on large, instruction-tuned models (GPT-4-class, Claude, PaLM-540B, large InstructGPT). Below roughly 10B parameters the gains thin out fast.

Escalation. If zero-shot CoT still misses, add a few worked examples (few-shot CoT), wrap it in self-consistency (sample many chains, majority-vote the answer), or switch to Plan-and-Solve, which front-loads an explicit plan to cut missing-step errors.

The trigger phrase matters

The exact wording is load-bearing. Kojima et al. ran a robustness study over alternative triggers on MultiArith; Let's think step by step. won among hand-written instructive phrases, while misleading or empty triggers ("Don't think, just answer") performed far worse. Some phrasings that worked nearly as well:

Let's think step by step.                              # the canonical trigger
Let's think about this logically.
Let's solve this problem by splitting it into steps.
First, let's think about this carefully.

Later, the APE method (Zhou et al., 2022) searched automatically for a better trigger and found Let's work this out in a step by step way to be sure we have the right answer., which nudged text-davinci-002 from 78.7% to 82.0% on MultiArith and from 40.7% to 43.0% on GSM8K. The lesson: the trigger is a tiny prompt you can optimize, not a magic incantation.

Structure and components

A complete zero-shot CoT prompt has three parts:

• The question — stated plainly, exactly as you'd ask without CoT.
• The reasoning trigger — the instruction that flips the model into step-by-step mode. Required; this is the whole technique.
• The answer-extraction cue (stage two) — a phrase that forces a clean, parseable answer out of the free-form reasoning. Optional for human reading, important for programmatic parsing.

# Stage 1 — reasoning extraction
{question}
A: Let's think step by step.

# Stage 2 — answer extraction (re-feed Stage 1 output, then append)
{question}
A: Let's think step by step.
{generated_reasoning}
Therefore, the answer (arabic numerals) is

Tune the extraction cue to the answer type: the answer (arabic numerals) is for math, the answer is (Yes or No) for booleans, among A through E, the answer is for multiple choice.

Core algorithm

The whole technique in a dozen lines — two model calls glued by string concatenation:

def zero_shot_cot(question, model, extract_cue="Therefore, the answer is"):
    # Stage 1: elicit reasoning
    reason_prompt = f"Q: {question}\nA: Let's think step by step."
    reasoning = model.generate(reason_prompt, max_tokens=512, temperature=0)

    # Stage 2: extract a parseable answer conditioned on the reasoning
    answer_prompt = f"{reason_prompt}{reasoning}\n{extract_cue}"
    answer = model.generate(answer_prompt, max_tokens=32, temperature=0)
    return answer.strip(), reasoning

Keep temperature=0 for deterministic single-answer use; raise it only if you're sampling multiple chains for self-consistency.

Configuration

Implementation workflow

1. Start from your plain zero-shot prompt and confirm it underperforms.
2. Append the trigger and run stage one; eyeball the reasoning chains.
3. Add a stage-two extraction cue matched to your answer type.
4. Build a parser for that answer format and test it on a holdout set.
5. Measure against the no-CoT baseline; keep CoT only if the lift beats the added token cost.

Do and don't

• Do put the trigger right where the answer would start (after A:), not buried in a system preamble.
• Do separate reasoning from answer extraction when you parse outputs programmatically.
• Don't use it on single-step tasks — it wastes tokens and can degrade accuracy by overthinking.
• Don't parse the answer out of the raw reasoning with brittle regex; use the extraction cue.
• Don't expect gains on small models; the ability is emergent at scale.

Debugging

Limitations

• Lower ceiling than few-shot CoT. When you can supply good worked examples, few-shot CoT usually wins; zero-shot CoT closes much of the gap but rarely beats it on hard tasks.
• Answer extraction needs task-specific cues. The reasoning generalizes, but pulling a clean answer out often requires a hand-tuned extraction phrase per format, limiting true plug-and-play use.
• Emergent, not universal. It needs a large, capable model. Small models barely benefit.
• Unfaithful reasoning risk. A plausible-looking chain can still reach a wrong answer, or rationalize a guess after the fact — the steps aren't a guarantee of correctness.

Advanced techniques

• Self-consistency on top. Sample many zero-shot CoT chains at higher temperature and majority-vote the final answers. This is the single biggest accuracy add-on.
• Plan-and-Solve. A zero-shot successor that asks the model to first devise a plan, then execute it, specifically to reduce missing-step and calculation errors.
• Auto-CoT. Uses zero-shot CoT to auto-generate reasoning chains, clusters questions for diversity, then assembles them into few-shot demonstrations — bridging zero-shot and few-shot.
• Trigger optimization. Treat the trigger as a searchable prompt (as APE did) and tune it against your own benchmark.

Risk and ethics

Ecosystem and related techniques

Zero-shot CoT sits at the root of the reasoning-prompt family. Few-shot CoT is its example-driven sibling; self-consistency, Tree of Thoughts, and Plan-and-Solve are descendants that add sampling, search, or planning. It composes cleanly with retrieval (reason over retrieved context) and with agents (the reasoning step before an action). Most LLM frameworks expose it as nothing more than a prompt template, since that's all it is.

Future directions

Reasoning-tuned and "thinking" models increasingly do step-by-step deliberation by default, folding zero-shot CoT into the model itself. The open questions are about faithfulness — making the chain actually reflect the computation — and about when explicit reasoning helps versus when it just burns tokens. Adaptive routing that invokes CoT only on hard inputs is an active frontier.

Summary

• Zero-shot CoT adds one phrase — "Let's think step by step" — to elicit step-by-step reasoning with no examples (Kojima et al., 2022).
• It mechanically works in two stages: generate the reasoning, then extract a parseable answer from it.
• Verified gains on text-davinci-002: MultiArith 17.7% to 78.7%, GSM8K 10.4% to 40.7%.
• It's emergent — it pays off on large instruction-tuned models and barely helps small ones.
• Use it for multi-step reasoning with no examples on hand; skip it for single-step recall or classification.
• The trigger phrase is optimizable: APE's variant reached 82.0% on MultiArith and 43.0% on GSM8K.
• Few-shot CoT usually has a higher ceiling; self-consistency and Plan-and-Solve are the natural next steps.