Consistency-based self-adaptive prompting (COSP): a complete guide

Few-shot prompting works, but it needs hand-written examples and ground-truth labels. COSP gets you the same lift with neither: it has the model answer a handful of unlabeled questions, keeps the answers it agreed with itself on, and feeds those back as its own examples. In Wan et al.'s ACL 2023 paper, this self-built prompt improved accuracy up to 15% over zero-shot and matched or beat few-shot across a range of reasoning tasks.

The bet is simple. When a model reaches the same answer down several different reasoning paths, that answer is usually right. COSP turns that hunch into a score, picks the most consistent self-generated examples, and uses them to prompt the real test questions.

See it work

Take a grade-school math question. Plain zero-shot Chain-of-Thought samples it a few times and the answers scatter:

Q: A store has 45 apples. It sells 12 in the morning and 18 in the
   afternoon. How many remain?
A: Let's think step by step. ...  → 15
A: Let's think step by step. ...  → 15
A: Let's think step by step. ...  → 27   (subtracted only the morning)
A: Let's think step by step. ...  → 15

Three of four runs agree on 15. Low disagreement means low entropy, which means the model is confident — so COSP keeps this question, its best rationale, and the answer 15 as a pseudo-demonstration. Repeat over a pool of unlabeled questions, select the most consistent few, and the test prompt now leads with examples the model itself is sure about:

Q: A store has 45 apples. It sells 12 in the morning and 18 in the
   afternoon. How many remain?
A: Start with 45. Morning: 45 - 12 = 33. Afternoon: 33 - 18 = 15.
   The answer is 15.

Q: {the actual test question}
A: Let's think step by step.

No human wrote that example. No label confirmed it. The model bootstrapped it from its own confident output.

The mental model

Think of a study group with no answer key. You can't check who's right, but you can notice that four classmates independently landed on the same answer while one outlier disagreed. You'd trust the agreed answer and use that worked solution as your template. COSP is that instinct, automated.

Consistency is a free, label-free proxy for correctness — so let the model grade its own confidence and learn from the answers it's surest about.

How it works

COSP runs in two passes. The first builds demonstrations; the second uses them.

Stage 1 — build the demonstrations:

1. Gather a pool of unlabeled questions from the target domain.
2. For each question, sample m zero-shot CoT rationales at non-zero temperature.
3. Extract the final answer from each rationale and look at the spread per question.
4. Score every candidate: low normalized entropy across its answers is good; repeated, degenerate phrasing inside a rationale is penalized; diversity across selected questions is rewarded.
5. Keep the top k candidates — confident, clean, and varied.

Stage 2 — answer for real:

1. Prepend the selected demonstrations to each test question.
2. Sample several reasoning paths (self-consistency).
3. Take the majority-vote answer.

Why it works

Ranked by how much each lever moves the result:

Where it shines

COSP is built for reasoning tasks with discrete, comparable answers — the kind where "are these two outputs the same?" has a clear yes or no. Wan et al. evaluated it on six benchmarks across three models (PaLM-62B, PaLM-540B, and GPT-3 code-davinci-001):

• Arithmetic reasoning: MultiArith, GSM8K, AddSub, SingleEq.
• Commonsense reasoning: CommonsenseQA, StrategyQA.

The headline: up to 15% accuracy improvement over zero-shot CoT, while matching or exceeding few-shot baselines that do use labeled examples. Every model improved over zero-shot on every task — with one exception: GPT-3 on GSM8K, the hardest set, where the candidate pool from pure zero-shot was too weak. The fix was COSP-FS, which seeds Stage 1 with a few-shot prompt so the model generates better candidates; in that setting COSP outperformed both zero-shot CoT and 5-shot CoT.

The natural home is anywhere you have unlabeled questions but no labels: automated math tutoring, scientific calculations, yes/no compliance checks, code-output prediction, logic puzzles — tasks where answers can be verified for agreement even if you can't supply the truth up front.

When to use it (and when not)

Reach for COSP when answers are deterministic and extractable, zero-shot CoT underperforms, you have no labels but do have representative unlabeled questions, the task needs multi-step reasoning, and you can afford several samples per question.

Skip it when the task is open-ended or subjective (no consistency signal — use USP), you're on a native reasoning model like o1/o3 (it already reasons internally), you have hard real-time latency limits, you lack relevant unlabeled questions, or you're on a small model that can't reason coherently.

Model fit: like all CoT methods, COSP needs a model big enough to reason — the paper's smallest was PaLM-62B. Tiny or non-instruction-tuned models generate incoherent chains that no amount of selection can rescue.

Variants and alternatives:

Structure and components

Required pieces: an unlabeled question pool, a zero-shot CoT trigger ("Let's think step by step"), an answer extractor, a scoring function (entropy plus repetition penalty), a top-k selector with a diversity check, and majority voting at inference.

Optional pieces: the few-shot bootstrap of COSP-FS, the adaptive demo count of COSP+, and domain-specific answer extraction or repetition detection.

The selection score is the heart of it. For each question, compute the normalized entropy of its sampled answers (consistency), penalize repetitive rationales, and prefer a diverse final set:

import numpy as np
from collections import Counter

def normalized_entropy(answers):
    """Low entropy = answers agree = model is confident."""
    if len(answers) <= 1:
        return 0.0
    counts = Counter(answers)
    probs = [c / len(answers) for c in counts.values()]
    h = -sum(p * np.log(p) for p in probs if p > 0)
    return h / np.log(len(answers))  # 0 (unanimous) .. 1 (uniform)

def score(answers, repetition, trade_off=0.2):
    """Lower is better: confident and non-degenerate."""
    return normalized_entropy(answers) + trade_off * repetition

def select_demos(candidates, k):
    """candidates: list of dicts with question, rationale, answers, repetition."""
    for c in candidates:
        c["score"] = score(c["answers"], c["repetition"])
    ranked = sorted(candidates, key=lambda c: c["score"])
    chosen, seen = [], set()
    for c in ranked:                       # greedy diversity: one demo per question
        if c["question"] not in seen:
            chosen.append(c)
            seen.add(c["question"])
        if len(chosen) >= k:
            break
    return chosen

The prompt the selected demos build for Stage 2 is plain few-shot CoT:

Q: {selected_question_1}
A: {selected_rationale_1} The answer is {selected_answer_1}.

Q: {selected_question_2}
A: {selected_rationale_2} The answer is {selected_answer_2}.

Q: {test_question}
A: Let's think step by step.

Configuration

Tuning notes: arithmetic likes higher m for better coverage and a lower temperature for coherent math; commonsense tolerates higher temperature for diversity. For hard multi-step tasks, switch to COSP-FS and raise k before anything else.

Implementation workflow

A single Stage 2 inference function — sample, extract, majority-vote — is the part you call per request:

from collections import Counter

def cosp_inference(test_question, demos, s=5, temperature=0.7):
    prompt = "".join(
        f"Q: {d['question']}\nA: {d['rationale']}\n\n" for d in demos
    ) + f"Q: {test_question}\nA: Let's think step by step."

    answers = []
    for _ in range(s):
        text = call_llm(prompt, temperature=temperature)   # your provider here
        answers.append(extract_answer(text))
    return Counter(answers).most_common(1)[0][0]           # majority vote

The end-to-end flow:

1. Collect 20–50 representative unlabeled questions.
2. Set n, m, k, s, temperature, λ from the table.
3. Run Stage 1 offline: generate candidates, score, select demos.
4. Eyeball the selected demonstrations — are they actually correct and clean?
5. Run Stage 2 on a small test sample, measure gain over zero-shot.
6. Cache the demos and serve; re-generate when the domain or model changes.

Do:

• Pre-compute and cache Stage 1 — it's the expensive part and it's reusable.
• Validate your answer extractor before the full run; bad extraction wrecks entropy.
• Keep the unlabeled pool diverse so demos cover different problem types.
• Hold out your test questions — never select demos from them.

Don't:

• Apply it to open-ended or subjective tasks; the consistency signal isn't there.
• Drop the repetition penalty; it's what filters degenerate, looping rationales.
• Skimp on the question pool; too few candidates means weak demos.
• Assume demos transfer across models or domains — re-run Stage 1.

Debugging

• Low accuracy → inspect the selected demos. If they're wrong, raise n/m or switch to COSP-FS; if extraction is off, fix the parser; if the task has no deterministic answer, COSP is the wrong tool.
• Unstable across runs → lower Stage 2 temperature (0.3–0.5) and raise s to 7–10 for a firmer majority.
• Demos look degenerate → questions may be too hard for zero-shot (use COSP-FS), or the repetition/embedding scoring is broken — verify it discriminates.
• Entropy is zero everywhere → all samples agree, possibly wrongly; raise temperature so disagreement can surface, and sanity-check against another method.

How to prove it beats the baseline

Run COSP against plain zero-shot CoT on the same held-out set and compare accuracy — the gain should clear a few points to be worth the cost:

def evaluate(test_set, demos, s=5):
    cosp_correct = zs_correct = 0
    for q, truth in test_set:
        if cosp_inference(q, demos, s) == truth:
            cosp_correct += 1
        if zero_shot_cot(q) == truth:           # baseline: no demos, single pass
            zs_correct += 1
    n = len(test_set)
    return {"cosp": cosp_correct / n, "zero_shot": zs_correct / n}

Also ablate the pieces — with and without the repetition penalty, with and without diversity — to confirm each is earning its keep, and watch the entropy distribution: if it has no variance, the score can't discriminate and m is too small.

Limitations

• Reasoning-only. Consistency needs comparable answers. Open-ended generation and subjective tasks break the entire selection mechanism — that's what USP is for.
• Confidently wrong. The method assumes consistent means correct. Systematic biases and common misconceptions produce low-entropy wrong answers that sail through selection.
• Compute overhead. The extra sampling in both stages is inherent to the approach; you can cache Stage 1 but you can't remove the sampling.
• Needs scale. Below roughly 100B parameters, reasoning chains are too incoherent for selection to help.
• Cold start. No representative unlabeled questions, no demos. Mismatched questions yield irrelevant demos.
• No feedback loop. Standard COSP is two-stage and fixed — a bad Stage 1 propagates straight into Stage 2 with no correction.

Advanced techniques

You can sharpen the demonstrations COSP writes for itself. Keeping rationales consistently formatted, with explicit calculation steps and an unambiguous final-answer line, makes them better templates. Adding a short verification step ("Let me check: 15 + 12 + 18 = 45") to demos can prime the model to self-check on the test question.

It also composes with other techniques. Pair it with RAG — retrieve context, then answer with COSP demos. Feed its output into a self-refinement pass. Or run it iteratively, using the demos from one round to bootstrap better candidates in the next. The entropy scores are reusable too: high-entropy questions are exactly the ones worth sending to a human for labels, which folds COSP neatly into an active-learning loop.

Risks and ethics

There's also a prompt-injection surface: unlabeled questions are model inputs, so sanitize them before they enter Stage 1. And because selection is automatic, the reasoning that ends up steering your outputs is opaque — log which demos were chosen and version them so you can trace a bad result back to its cause.

Ecosystem and related techniques

COSP sits on top of three earlier ideas and one later one:

No major framework ships COSP as a one-liner, but it's straightforward to build as a custom chain in LangChain or a module in DSPy — Stage 1 is a generate-score-select loop, Stage 2 is self-consistency over a templated prompt.

Transitions: moving from zero-shot, collect production queries as your unlabeled pool, run Stage 1, then A/B the two. Moving on from COSP, its high-confidence demonstrations make decent fine-tuning data once you've accumulated enough — or you graduate to a native reasoning model and drop the scaffolding entirely.

Future directions

The open questions are whether a learned scoring function beats hand-tuned entropy-plus-repetition, whether demonstrations transfer between related tasks, and how performance scales with n, m, and k. The live frontier is native reasoning: as models like o1/o3 internalize consistency during training, external COSP may become unnecessary at the top end — but the principle of grading your own confidence and learning from it stays useful wherever labels are scarce.

Summary

• What: COSP builds few-shot demonstrations automatically from a model's own confident zero-shot outputs — no hand-written examples, no labels.
• Why: Consistency (low answer entropy) is a free, label-free proxy for correctness, so the model can pick its own best examples.
• How: Stage 1 samples zero-shot CoT over unlabeled questions, scores by entropy plus a repetition penalty with a diversity check, and keeps the top k; Stage 2 prepends them and answers by self-consistency majority vote.
• When: Reasoning tasks with discrete, comparable answers and no labels; skip it for open-ended, subjective, or latency-critical work, and on tiny models.
• Where: Arithmetic and commonsense reasoning — MultiArith, GSM8K, AddSub, SingleEq, CommonsenseQA, StrategyQA — and any domain with unlabeled questions but no truth.
• Which: Standard COSP for usable candidates, COSP-FS for hard tasks like GSM8K, COSP+ for mixed difficulty, USP once you leave reasoning behind.
• Evidence: Wan et al., "Better Zero-Shot Reasoning with Self-Adaptive Prompting," Findings of ACL 2023 — up to 15% over zero-shot, matching or exceeding few-shot across three LLMs.

References

• Wan, X., Sun, R., Dai, H., Arik, S. O., & Pfister, T. (2023). Better Zero-Shot Reasoning with Self-Adaptive Prompting. Findings of ACL 2023. arXiv:2305.14106
• Kojima, T., et al. (2022). Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.
• Wang, X., et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
• Zhang, Z., et al. (2022). Automatic Chain of Thought Prompting in Large Language Models. ICLR 2023.
• Wan, X., et al. (2023). Universal Self-Adaptive Prompting (USP). EMNLP 2023.
• Google Research blog: Zero-shot adaptive prompting