Uncertainty-routed chain-of-thought: a complete guide

Sampling many chain-of-thought answers and taking the majority vote usually helps, but on the hardest questions the chains scatter and the vote picks a confident wrong answer. Uncertainty-routed CoT fixes that with one move: trust the majority vote only when the chains agree, and fall back to the plain greedy answer when they don't. Google used it to push Gemini Ultra on MMLU from 84.0% (greedy) to 90.0% with 32 samples, while plain majority voting reached only 85.0% (Gemini technical report, 2023).

See it work

Same model, same 32 chain-of-thought samples per question. The only difference is how you pick the final answer.

Take a hard question where the model is unsure. The 32 chains barely agree:

Question (hard): A,B,C,D multiple choice
32 CoT samples vote: A=7  B=8  C=9  D=8   (top answer wins 9/32 = 28%)
Greedy (single most-likely answer): B   ← correct

Plain self-consistency → picks C (the 28% plurality)  ✗
Uncertainty-routed → agreement 28% is below threshold → defer to greedy → B  ✓

Now an easy question where the model is sure. The chains pile up on one answer:

Question (easy): A,B,C,D multiple choice
32 CoT samples vote: A=1  B=0  C=1  D=30   (top answer wins 30/32 = 94%)
Greedy (single most-likely answer): D

Plain self-consistency → picks D  ✓
Uncertainty-routed → agreement 94% is above threshold → trust the vote → D  ✓

Routing only changes the verdict on the shaky questions, exactly where blind majority voting hurts. On confident questions it behaves like normal self-consistency.

The mental model

Think of a jury that only delivers a verdict when it's near-unanimous. If the jurors split evenly, you don't average their guesses; you defer to the one expert whose single call you trust most. The model's many sampled chains are the jurors; its greedy answer is the expert.

Vote when the chains agree. When they don't, stop voting and take the single most-likely answer instead.

How it works

You run two decoding strategies and let confidence decide which one wins per question.

1. Sample k chains. Generate k chain-of-thought completions at non-zero temperature. Gemini used k = 32.
2. Tally the votes. Extract the final answer from each chain and count how often each answer appears.
3. Measure confidence. Use the agreement of the top answer (its share of the k votes) as a confidence signal.
4. Route. If agreement clears a threshold, output the majority vote. If it doesn't, discard the vote and output the greedy (temperature-0, maximum-likelihood) answer instead.
5. Tune the threshold. Pick the threshold per model on a validation split, choosing the value that maximizes held-out accuracy.

Why it works

The whole gain comes from one observation: majority voting and greedy decoding fail on different questions, so picking the right one per question beats committing to either.

Where it shines

The technique was built for and proven on multiple-choice reasoning benchmarks, where every answer is a discrete option you can vote over.

• Massive multitask knowledge tests. On MMLU, Gemini Ultra scored 84.0% with greedy decoding, 85.0% with plain 32-sample CoT majority voting, and 90.0% with uncertainty-routed CoT@32. Plain voting barely moved the needle; routing added six points.
• Cross-model gains. Using the same method, GPT-4 improved from 84.2% greedy to 87.3% with uncertainty-routed CoT@32 on MMLU, so the benefit isn't unique to one model family.
• Any task with a votable answer. Anywhere you can extract a discrete answer (a label, a multiple-choice letter, a final number) and run majority voting, routing can protect you from the cases where voting backfires.

When to use it (and when not)

Reach for it when:

• The answer is discrete and votable (multiple choice, classification, a final numeric answer).
• You can afford to sample many chains per question (offline eval, benchmarking, high-stakes single calls).
• You have a validation split to tune the confidence threshold.
• You're already running self-consistency and want to stop it from hurting on hard items.

Skip it when:

• The output is open-ended prose where there's no clean answer to count votes over.
• Latency or cost rules out sampling k chains at all — fall back to a single greedy answer.
• You have no held-out data to set the threshold, so plain self-consistency is the safer default.

Model fit. It pays off most on capable models whose chains are usually right but occasionally scatter; both Gemini Ultra and GPT-4 gained on MMLU. A weak model whose chains are noisy everywhere has no reliable confident regime for voting to exploit.

Escalation path. Start with greedy CoT. If accuracy stalls and you can afford samples, add self-consistency (plain majority voting). If voting helps overall but hurts on a slice of hard questions, add uncertainty routing on top.

Structure and components

Three pieces have to be in place:

• A sampler. Produces k chain-of-thought completions at non-zero temperature, so the chains vary.
• A greedy decoder. Produces the single maximum-likelihood answer at temperature 0, used as the fallback.
• A confidence function and threshold. Converts the k votes into a confidence score (top answer's vote share) and compares it to a tuned threshold to choose the route.

The answer extractor is the quiet dependency: every chain must yield a comparable final answer, or the votes can't be tallied. Multiple-choice letters and short final answers extract cleanly; free-form paragraphs don't.

The core algorithm

The whole method is a routing wrapper around two decoders you already have.

from collections import Counter

def uncertainty_routed_cot(question, model, k=32, threshold=0.5, temperature=0.7):
    # 1. Sample k chain-of-thought answers.
    samples = [model.cot(question, temperature=temperature) for _ in range(k)]
    answers = [extract_answer(s) for s in samples]

    # 2. Tally votes and measure confidence as the top answer's share.
    counts = Counter(answers)
    top_answer, top_votes = counts.most_common(1)[0]
    confidence = top_votes / k

    # 3. Route: trust the vote only if confident, else defer to greedy.
    if confidence >= threshold:
        return top_answer            # majority vote
    return extract_answer(model.cot(question, temperature=0.0))   # greedy

The threshold isn't guessed. You sweep it on a validation set and keep the value that maximizes accuracy:

def tune_threshold(val_questions, val_labels, model, k=32, grid=None):
    grid = grid or [i / 20 for i in range(1, 20)]   # 0.05 .. 0.95
    best_t, best_acc = None, -1.0
    for t in grid:
        preds = [uncertainty_routed_cot(q, model, k=k, threshold=t) for q in val_questions]
        acc = sum(p == y for p, y in zip(preds, val_labels)) / len(val_labels)
        if acc > best_acc:
            best_t, best_acc = t, acc
    return best_t

Configuration

Implementation workflow

1. Wire an answer extractor that turns any chain into a comparable final answer.
2. Build the sampler (k chains, non-zero temperature) and the greedy decoder (temperature 0).
3. On a validation split, sweep the threshold and lock in the value with the best held-out accuracy.
4. Deploy the router with that threshold; log per-question confidence so you can audit routing decisions.
5. Re-tune the threshold whenever you change models, k, or the task — the crossover point moves.

Do

• Tune the threshold on held-out data, per model.
• Keep the sampling temperature high enough that chains actually diverge.
• Log the confidence score and which route fired, for debugging and audits.

Don't

• Reuse one model's threshold on another model.
• Apply it to open-ended outputs you can't vote over.
• Assume more samples always help — past a point, confidence estimates barely sharpen while cost climbs.

Debugging

Testing and how to prove it

Run all three strategies on the same held-out set and compare accuracy directly. The point you must demonstrate is that routing beats both greedy and plain voting — otherwise you're just paying for self-consistency.

def compare(test_q, test_y, model, threshold, k=32):
    greedy = [extract_answer(model.cot(q, temperature=0.0)) for q in test_q]
    voting = [majority_vote(model, q, k) for q in test_q]
    routed = [uncertainty_routed_cot(q, model, k=k, threshold=threshold) for q in test_q]
    acc = lambda preds: sum(p == y for p, y in zip(preds, test_y)) / len(test_y)
    return {"greedy": acc(greedy), "self_consistency": acc(voting), "routed": acc(routed)}

Slice the results by confidence: routing should match voting on high-agreement questions and match (or beat) greedy on low-agreement ones. If it doesn't, your threshold is mis-tuned.

Limitations

• It needs a votable answer. No discrete answer means no vote, so the method doesn't apply to open-ended generation.
• It inherits self-consistency's cost. k generations per question is expensive; routing adds the greedy call on top.
• The threshold is task- and model-specific. It has to be re-tuned whenever those change, and a bad threshold can erase the gains.
• Top-vote share is a coarse confidence signal. It ignores how the runner-up answers are distributed, so it can misjudge confidence on near-ties.
• Weak models gain little. If chains are noisy everywhere, there's no confident regime for voting to exploit.

Advanced variations

The routing idea generalizes beyond a single hard threshold:

• Richer confidence signals. Replace top-vote share with entropy over the vote distribution, or with sequence log-probabilities, for a finer uncertainty estimate.
• Three-way routing. Add a middle band that triggers more samples (escalate k) before deciding, instead of a binary vote-or-greedy choice.
• Per-category thresholds. Tune separate thresholds for question types whose difficulty differs, rather than one global threshold.

Risk and ethics

Ecosystem

Uncertainty-routed CoT sits squarely in the chain-of-thought family. It builds directly on self-consistency, which builds on chain-of-thought prompting. Its closest neighbor is adaptive-consistency, which also uses confidence — but to decide when to stop sampling, lowering average cost, rather than to decide which decoder to trust. The two are complementary: you can stop early once confident, then route.

Hybrids that fit naturally: pair routing with retrieval (so the chains reason over grounded context), or use it as the answer-selection layer on top of any technique that emits a votable final answer.

Future directions

The open questions are mostly about the confidence signal. Better-calibrated uncertainty estimates — beyond raw vote share — would let routing trigger correctly on more questions and could shrink k. Learned routers that decide per question whether voting is even worth the samples point toward spending compute only where it changes the answer.

Summary

• Uncertainty-routed CoT samples k chain-of-thought chains, then trusts the majority vote only when the chains agree above a tuned threshold; otherwise it defers to the greedy answer.
• It exists because plain majority voting can pick a confident wrong answer on hard, inconsistent questions — exactly where the single most-likely answer does better.
• On MMLU it took Gemini Ultra from 84.0% (greedy) and 85.0% (plain CoT@32 voting) to 90.0% (routed CoT@32); GPT-4 went from 84.2% to 87.3% with the same method.
• Use it for discrete, votable answers when you can afford k samples and have a validation split to tune the threshold; skip it for open-ended text or when you can't sample.
• It costs about kx a single inference and re-needs threshold tuning per model and task, but it never underperforms the better of greedy and voting once tuned.