Automatic chain-of-thought (Auto-CoT): a complete guide

Hand-writing chain-of-thought examples works, but it doesn't scale — every new task needs a fresh batch of question-reasoning-answer triples, crafted by someone who knows both the domain and the model. Auto-CoT removes the human. It clusters your questions for diversity, lets the model write its own reasoning chains with "Let's think step by step," and assembles them into a few-shot prompt. The payoff: it matches or beats hand-crafted Manual-CoT on all 10 benchmark reasoning tasks (47.9% vs 46.9% on GSM8K, 92.0% vs 91.7% on MultiArith with GPT-3), with zero manual effort (Zhang et al., ICLR 2023, arXiv 2210.03493).

See it work

Say you're building a math tutor and you want eight worked examples to prime the model. The obvious move is to grab the eight questions most similar to whatever the student just asked. That's the trap Auto-CoT was designed around.

Similar questions share failure modes. If the model fumbles the arithmetic on one, it fumbles all eight the same way — correlated errors, no safety net:

Test question: "A train goes 60 mph for 2.5 hours. How far?"

Retrieval (similar) picks 8 near-identical rate problems.
Model mis-multiplies on one → likely mis-multiplies on all 8.
The demonstrations reinforce the SAME mistake.

Auto-CoT picks one question from each cluster instead — one rate problem, one counting problem, one unit conversion, and so on. A flaw in any single demonstration gets diluted by the correct ones around it. And it writes each chain itself:

Q: Roger has 5 tennis balls. He buys 2 cans of tennis balls.
   Each can has 3 balls. How many does he have now?
A: Let's think step by step. Roger started with 5 balls.
   He bought 2 cans of 3 balls each, so 2 × 3 = 6.
   5 + 6 = 11. The answer is 11.

No one wrote that demonstration. The model did, from a bare question plus "Let's think step by step." Repeat across eight diverse clusters and you have a few-shot prompt that cost nothing to build.

The mental model

Think of a teacher assembling a worked-example sheet before an exam. A lazy teacher photocopies eight versions of the same problem — if their solution has a typo, every example is wrong. A good teacher pulls one problem from each chapter, so a single slip can't poison the whole sheet. Auto-CoT is the good teacher, and clustering is how it finds "one per chapter" automatically.

Diversity beats similarity: spread your examples across reasoning types so no single error pattern can dominate.

How it works

Auto-CoT runs in two phases. Demonstration construction happens once per dataset (offline); inference is then ordinary few-shot CoT (online).

1. Encode. Turn every question into a dense vector with Sentence-BERT, so semantically similar questions sit close together.
2. Cluster. Run k-means with k equal to the number of demonstrations you want (default k=8). Each cluster is a different semantic region of the question space.
3. Sample. Within each cluster, sort questions by distance to the centroid and walk outward from the most representative one.
4. Generate. For a candidate question, append "Let's think step by step" and let the model write a zero-shot reasoning chain.
5. Filter. Accept the chain only if it clears simple heuristics: question ≤ 60 tokens, rationale ≤ 5 steps, and (for arithmetic) the answer appears in the rationale. If it fails, move to the next question in the cluster.
6. Assemble and infer. Concatenate the k accepted demonstrations into one few-shot prompt, append the test question, and run the model normally. Inference is single-pass — no test-time iteration.

If no question in a cluster passes the filters, the centroid question is used with its generated chain regardless, so every cluster always contributes one demonstration.

Why it works

The empirical effect breaks down roughly like this (ranked by impact in the original ablations):

The deeper reason is statistical. If each demonstration is correct with probability p and diversity makes errors independent, the chance that most of k demonstrations are correct follows the binomial. With the paper's empirical p ≈ 0.875 and k=8, you expect 7 correct demonstrations out of 8 — and the diverse, uncorrelated wrong one barely moves the needle. That same independence is why Auto-CoT held its accuracy even when up to 50% of demonstrations contained incorrect reasoning, while retrieval-based sampling collapsed under the same conditions because its errors compounded.

Where it shines

Auto-CoT matches or exceeds Manual-CoT on all 10 benchmarks the paper tested, using GPT-3 (text-davinci-002):

The biggest gains land on Coin Flip (+2.7%), AddSub (+3.5%), and GSM8K (+1.0%). Arithmetic word problems are the sweet spot — diverse demonstrations capture addition, multiplication, multi-step, and unit-conversion patterns without anyone curating them. Commonsense (StrategyQA, CSQA) and symbolic tasks (Last Letter, Coin Flip) also benefit wherever questions split cleanly into reasoning sub-types.

The story holds on a code-trained model too. With Codex (code-davinci-002), Auto-CoT actually beat Manual-CoT on GSM8K (62.8% vs 59.4%, +3.4%) and AddSub (91.9% vs 84.6%, +7.3%), trailing only on MultiArith (93.2% vs 96.8%).

Averaged across all 10 tasks, the ladder of effort-vs-accuracy looks like this:

Beyond benchmarks, the clustering-then-generate pattern transfers to any task with natural sub-types: auto-generating worked examples across a curriculum, clustering support tickets by type and explaining the routing logic, covering diverse debugging scenarios in code review, or spanning causal/correlational/experimental patterns in scientific reasoning.

When to use it (and when not)

Reach for Auto-CoT when:

• You want few-shot CoT accuracy without paying for manual demonstration design.
• You're deploying across many tasks and need task-adaptive demonstrations.
• Your dataset is big enough for meaningful clustering (30+ questions, ideally 100+) and spans multiple reasoning sub-types.
• Your model is strong enough to write reasonable zero-shot chains, and few-shot CoT already beats zero-shot CoT on the task.

Skip it when:

• You're on a native reasoning model (o1, o3, Gemini 2.5 thinking mode) — external demonstrations interfere with built-in reasoning.
• Zero-shot CoT already saturates the task, so demonstrations add nothing.
• You have fewer than 20 questions — clustering is meaningless.
• The model's zero-shot CoT quality is too low for the domain (highly specialized medical or legal reasoning).
• You need per-instance adaptation rather than one fixed demonstration set.

Model fit. You need roughly 100B+ parameters for reliable zero-shot chain generation — think GPT-3.5/4, Claude 3+, PaLM 540B, or open models at 70B+. Below ~100B the generated chains are too illogical to help. The clustering stage needs Sentence-BERT separately, a lightweight ~110M-parameter encoder that runs locally on CPU. A useful trick: generate demonstrations with a strong model, then run cheap inference with a weaker one — the high-quality chains amortize across every call.

When to escalate. If you can afford targeted annotation on the hardest (highest-uncertainty) questions, move to Active-CoT (Diao et al., 2023). With labeled data for pruning and policy-gradient selection, use Automate-CoT (Shum et al., 2023). When uniform prompting causes high per-cluster accuracy variance, CDW-CoT adapts prompts per instance. When inference accuracy is critical and you can eat the latency, stack Self-Consistency on top.

Structure and components

A working Auto-CoT pipeline needs six pieces, all required:

• Question pool — questions from the target task (full training set, a subset, or a representative sample of historical queries).
• Sentence encoder — Sentence-BERT, turning questions into vectors where similar ones cluster.
• Clustering algorithm — k-means partitioning the pool into k groups, k = number of demonstrations.
• Zero-shot CoT generator — the LLM itself, prompted with "Let's think step by step," turning a bare question into a question-rationale-answer triple.
• Heuristic filters — the rules that reject overly long or complex chains (≤ 60 tokens, ≤ 5 steps, answer present).
• Demonstration concatenator — assembles the accepted demonstrations into one consistently formatted few-shot prompt.

Optionally, a task-instruction prefix ("Solve the following math problems step by step"), an explicit answer-format spec ("End with 'The answer is [X]'"), or the streaming Auto-CoT* module that refreshes demonstrations as more questions arrive.

The demonstrations follow a plain Q/A pattern, repeated k times before the test question:

Q: [question closest to centroid of cluster 1]
A: Let's think step by step. [reasoning chain]. The answer is [X].

Q: [question closest to centroid of cluster 2]
A: Let's think step by step. [reasoning chain]. The answer is [X].

... (k demonstrations, typically k=8)

Q: [test question]
A:

For high-complexity tasks, raise k to 10–12 and relax the step limit to 7–8. For domain-specific work, swap in a domain-tuned encoder and adjust the token limit to typical question lengths. For format-critical tasks, push formatting into the instruction prefix and verify it during filtering. If you have fewer questions than your target k, lower k to match.

Implementation

The whole technique fits in one readable function: encode, cluster, then for each cluster generate and filter until one demonstration sticks.

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans

def build_auto_cot(questions, generate_chain, k=8, max_q_tokens=60, max_steps=5):
    """Construct k diverse, self-generated CoT demonstrations."""
    encoder = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = encoder.encode(questions)
    kmeans = KMeans(n_clusters=k, random_state=42).fit(embeddings)

    demos = []
    for cid in range(k):
        idx = np.where(kmeans.labels_ == cid)[0]
        dists = np.linalg.norm(embeddings[idx] - kmeans.cluster_centers_[cid], axis=1)
        ordered = idx[np.argsort(dists)]          # representative first

        for i in ordered:
            q = questions[i]
            if len(q.split()) > max_q_tokens:     # heuristic: skip long questions
                continue
            chain = generate_chain(q)             # zero-shot CoT: "Let's think step by step"
            if len(chain.strip().split("\n")) <= max_steps:
                demos.append({"q": q, "a": chain})
                break
        else:                                     # fallback: use the centroid question
            q = questions[ordered[0]]
            demos.append({"q": q, "a": generate_chain(q)})
    return demos

def auto_cot_prompt(demos, test_question):
    body = "".join(f"Q: {d['q']}\nA: {d['a']}\n\n" for d in demos)
    return body + f"Q: {test_question}\nA:"

generate_chain is just a zero-shot CoT call (append "Let's think step by step," temperature 0). The official implementation lives at amazon-science/auto-cot (mirrored at cooelf/Auto-CoT) with full encoding, clustering, generation, filtering, and eval scripts for all 10 datasets.

Configuration

Workflow and do/don't

Collect questions, cluster with k=8, generate with heuristic filtering, spot-check two or three demonstrations for obvious errors, evaluate on a held-out set against the zero-shot-CoT baseline, tune k and thresholds if needed, then cache and deploy. The cached demonstration set is reusable across every test question for that dataset.

Do: cache demonstrations, validate a sample manually before deploying, keep k aligned with your Manual-CoT baseline for fair comparison, and start from default thresholds.

Don't: use Auto-CoT with native reasoning models; skip the heuristic filter (it drops demonstration error rates from roughly 31% to 15%); swap clustering for random sampling; set k too high for a small pool (degenerate one- or two-question clusters give no useful centroid); or assume generated demonstrations are correct — they're generated, not verified.

Debugging

The single most common mistake is reaching for retrieval-based (similarity) sampling — the exact anti-pattern Auto-CoT exists to avoid.

Testing

Reserve 20–30% of questions as a test set and build demonstrations only from the rest (or k-fold cross-validate on smaller datasets). Compare Auto-CoT against zero-shot-CoT, random-sampling CoT, and Manual-CoT on the same questions, same model, same parameters. Track accuracy as the headline metric, plus demonstration error rate (aim under 20%), cluster coverage (every cluster yields a valid demonstration), and accuracy standard deviation across k-means random seeds. Because clustering is seeded, run 3–5 seeds and report mean ± standard deviation; use a paired bootstrap or McNemar's test when comparing two demonstration sets.

Limitations

Some limits are inherent to the design:

• Bounded by zero-shot quality. Demonstrations can never be better than what the model writes zero-shot. If it can't reason about a topic unprompted, the demonstrations will be flawed.
• Semantic clustering is not reasoning clustering. Sentence-BERT groups by surface semantics, not by required reasoning pattern — two similar-looking questions may need different strategies. Later work (PA-CoT, 2024) targets exactly this gap.
• Static demonstrations. Once built, the set is fixed for every test question; it doesn't adapt to an individual instance's difficulty.
• No ground-truth verification. Quality rests entirely on heuristic proxies (length, step count), never on checking whether a chain is actually correct.

Edge cases follow from these. Out-of-distribution questions far from any centroid get all-irrelevant demonstrations and degrade to roughly zero-shot-CoT level. Extreme class imbalance (90% one type, 10% another) can hand 7 of 8 clusters to the dominant type, undermining diversity. Genuinely ambiguous questions, or clusters with contradictory reasoning (one rounds up, another down), feed the model conflicting signals it can't resolve. Silhouette scores from the clustering flag questions that fit no cluster well, and if k-means won't converge, k-medoids or hierarchical clustering are drop-in fallbacks. The saving grace is graceful degradation: in the worst case Auto-CoT falls back to zero-shot-CoT performance rather than below it, and the 50% error tolerance holds the line through a lot of demonstration noise.

Advanced techniques

A few extensions earn their cost:

• Better generation prompts. Seed chains with "Let's think step by step. First, let's identify what we know" instead of the bare trigger, and relax the step limit for genuinely long problems.
• Self-consistency at construction. Generate several candidate chains per cluster and keep the one whose answer most others agree with — more API calls, cleaner demonstrations.
• Cross-model generation. Use a strong model to write demonstrations, a cheap one to run inference.
• Auto-CoT* (bootstrap). Process questions in batches; promote correctly answered ones into candidate demonstrations for later batches, gradually replacing zero-shot-generated chains with verified ones.

It also composes: Auto-CoT + Self-Consistency samples N inference paths and votes (diversity in demonstrations and inference); Auto-CoT + RAG adds retrieved context alongside the demonstrations for knowledge-heavy tasks; Auto-CoT + verification (CoVe) runs each generated demonstration through a verification prompt and discards failures.

Risk and ethics

Two structural risks compound this. Diversity can't save you from a systematic error — if the model always applies a formula wrong, every cluster inherits the flaw. And in the Auto-CoT* bootstrap loop, an early wrong answer can become a demonstration for later batches, creating a self-reinforcing error cycle. Generated chains can also encode the model's training biases, and the length heuristics may systematically exclude questions from domains or languages where questions run longer. Demonstrations are machine-generated, which can sit poorly with audit requirements in regulated domains — downstream users may not realize the "examples" guiding the model were themselves written by a model.

Ecosystem

Auto-CoT sits in a dense neighborhood of CoT variants:

It builds directly on Manual-CoT (Wei et al., 2022) and Zero-Shot-CoT (Kojima et al., 2022) — using the latter as its chain generator — and pairs naturally with Self-Consistency (Wang et al., 2022) at inference time. Complexity-Based Prompting is a close cousin — it also selects demonstrations by a property, but uses reasoning complexity rather than diversity. Tooling-wise, sentence-transformers and scikit-learn cover encoding and clustering; DSPy's BootstrapFewShot teleprompter automates demonstration selection in a conceptually similar way; LangChain and Haystack can host the API calls. EleutherAI's lm-evaluation-harness handles benchmark scoring.

Transitioning in is low-risk: from Zero-Shot-CoT, build a demonstration set and adopt it if it improves a validation set by more than 2%; from Manual-CoT, keep your hand-written set as a baseline and switch if Auto-CoT matches or beats it, or run a hybrid (manual for the hardest types, Auto-CoT for the rest). In production, tag demonstration sets with dataset + model version + timestamp, monitor accuracy on a rotating validation set, keep previous sets for rollback, and re-construct whenever the model version changes, the question distribution shifts, or accuracy degrades.

Future directions

The frontier is about loosening Auto-CoT's two biggest assumptions — that semantic similarity tracks reasoning similarity, and that one fixed set fits every instance. CDW-CoT (2025, AAAI) builds prompts per test instance by distance to cluster centers, the current state of the art on adaptivity. PA-CoT (2024) clusters by reasoning pattern rather than question semantics. Self-improving variants extend Auto-CoT* by promoting verified correct answers into the demonstration pool over time, multi-model construction picks the chain with the highest cross-model agreement per cluster, and multimodal Auto-CoT extends clustering to text-plus-image inputs for visual reasoning. As native reasoning models spread, future variants will likely supply task context and format guidance rather than reasoning templates — the model already knows how to think; it just needs to know what you want.

Summary

• What it is: automatic construction of few-shot CoT demonstrations by clustering questions for diversity and letting the model write its own reasoning chains via zero-shot CoT.
• Why it works: diverse demonstrations make errors independent, so a wrong chain gets diluted instead of compounding — diversity (~40%) and zero-shot capability (~25%) dominate the effect.
• The evidence: matches or beats Manual-CoT on all 10 benchmarks (47.9% vs 46.9% on GSM8K), holds accuracy through 50% bad demonstrations, and costs only ~10–20 setup calls (Zhang et al., ICLR 2023).
• When to use it: large, diverse question pools where few-shot CoT beats zero-shot CoT and manual design doesn't scale; default k=8, ≤ 60-token questions, ≤ 5-step rationales.
• When not to: native reasoning models, tiny datasets (under 20 questions), saturated zero-shot tasks, or specialized domains the model can't reason about unprompted.
• Where it's going: instance-adaptive (CDW-CoT) and reasoning-pattern-aware (PA-CoT) successors that drop the fixed-set and semantic-clustering assumptions.