Contrastive chain-of-thought prompting: a complete guide

Plain chain-of-thought shows a model how to reason, but it never shows what not to do. Contrastive chain-of-thought (contrastive CoT) fixes that by pairing each correct reasoning chain with a wrong one, so the model learns the mistakes to avoid as well as the steps to follow. On GPT-3.5-Turbo, this lifted accuracy on seven reasoning benchmarks, from 69.2% to 79.0% on GSM8K and from 40.8% to 56.8% on Bamboogle, per Chia et al., "Contrastive Chain-of-Thought Prompting" (2023).

See it work

Here's a standard few-shot CoT demonstration. The model sees one worked example, then your real question.

Q: A robe takes 2 bolts of blue fiber and half that much white fiber.
   How many bolts in total?
A: There are 2 bolts of blue fiber. Half of 2 is 1, so 1 bolt of white.
   2 + 1 = 3. The answer is 3.

Q: Weng earns $12 an hour for babysitting. Yesterday she did 50 minutes.
   How much did she earn?
A:

Plain CoT teaches the shape of good reasoning, but the model still fumbles arithmetic ordering on harder problems. Contrastive CoT adds a second chain to the same example, clearly labelled as wrong:

Q: A robe takes 2 bolts of blue fiber and half that much white fiber.
   How many bolts in total?
Correct explanation: 2 bolts of blue fiber. Half of 2 is 1, so 1 bolt of
   white. 2 + 1 = 3. The answer is 3.
Wrong explanation: Half of 2 is 4, so 4 bolts of white. 2 + 4 = 6.
   The answer is 6.

Q: Weng earns $12 an hour for babysitting. Yesterday she did 50 minutes.
   How much did she earn?
Correct explanation:

The wrong chain makes a careless "half of 2 is 4" slip explicit. The model now has a concrete error to steer clear of, and it's measurably less likely to repeat that class of mistake on the target question.

The mental model

Think of a driving instructor. A good one doesn't only narrate the perfect lane change. They also point at the car that drifted without signalling and say "see that, don't do that." You learn faster from the contrast than from the ideal alone.

Plain CoT shows the model a good answer. Contrastive CoT shows it a good answer and a bad one, so the boundary between them becomes part of the lesson.

How it works

1. Start from normal CoT demonstrations. Pick a handful of examples with correct step-by-step rationales, exactly as you would for few-shot CoT.
2. Generate an invalid chain for each. Take the correct rationale and corrupt it so the reasoning is plausible-looking but wrong.
3. Label both chains. Present the valid one as the correct explanation and the invalid one as the wrong explanation, so the model can tell them apart.
4. Append the target question. End the prompt asking for the correct explanation, prompting the model to reason while sidestepping the demonstrated errors.
5. Optionally add self-consistency. Sample several chains and majority-vote the final answer; the gains stack on top of contrastive demonstrations.

Why it works

The paper's starting point is a surprising prior finding: chain-of-thought still helps even when the demonstration rationales are invalid. That means the model leans heavily on the surface form of reasoning, not its logical soundness, and plain CoT never tells it which patterns are wrong. Contrastive CoT supplies that missing signal.

Where it shines

Contrastive CoT is a general enhancement to CoT, so it helps most where CoT already helps: multi-step problems with an objective answer. The paper reports consistent gains across arithmetic and factual question answering on GPT-3.5-Turbo (0301), using greedy decoding.

The improvement runs from about +4 points (AQuA) to +16 points (Bamboogle). Layering self-consistency on top widens the margin further: with self-consistency decoding the paper reports contrastive CoT reaching 86.2 on GSM8K and 89.6 on ASDiv, double-digit gains over self-consistent plain CoT on every benchmark tested.

When to use it (and when not)

Reach for it when:

• The task needs multi-step reasoning and plain CoT already gets you partway.
• The failures you see are reasoning slips (wrong operation, dropped entity), not missing facts.
• You can write or auto-generate a believable wrong chain for each demonstration.
• You have room in the context window for longer demonstrations.

Skip it when:

• The task is single-step classification or extraction, where CoT itself adds little.
• Failures come from missing knowledge; retrieval or a stronger model helps more than a contrast.
• Context budget is tight, since every demonstration now carries two chains.
• You can't construct a plausible wrong chain; an obviously absurd one teaches nothing.

Model fit. This was demonstrated on an instruction-tuned model in the GPT-3.5-Turbo class. It assumes a model already capable of following CoT; on very small models that can't sustain a reasoning chain, fix that first. Stronger models still benefit, but the headroom shrinks as plain CoT accuracy climbs.

Escalation. If contrastive CoT alone isn't enough, add self-consistency (sample many chains, majority-vote). If the bottleneck is knowledge rather than reasoning, escalate to retrieval-augmented generation instead.

Anatomy of a contrastive prompt

Each demonstration has three parts, and the target question reuses the same labels so the model knows which to produce.

Question: <the problem>
Correct explanation: <valid step-by-step chain> The answer is <X>.
Wrong explanation: <invalid step-by-step chain> The answer is <Y>.

... repeat for each demonstration ...

Question: <the target problem>
Correct explanation:

The labels matter. The valid chain is explicitly the "correct explanation" and the invalid one the "wrong explanation"; the model needs that framing to treat the second chain as a thing to avoid rather than a second example to imitate.

Building the invalid demonstration

The interesting question is how to make a good wrong chain. The paper grounds this in the idea that a rationale has two components: bridging objects (the key entities and numbers threaded through the steps) and the language template (the connective reasoning text). You can corrupt either, along two axes, coherence and relevance, giving four flavours of invalid rationale:

• Incoherent objects — keep the language, shuffle the entities and numbers so they no longer line up.
• Incoherent language — keep the objects, scramble the reasoning steps out of logical order.
• Irrelevant objects — swap in entities and numbers unrelated to the problem.
• Irrelevant language — replace the reasoning text with language unrelated to the problem.

The paper finds incoherent chains, especially shuffled bridging objects, the most useful contrast, and they automate it. The automatic method takes a correct rationale, pulls out its bridging objects, and randomly permutes them to produce a plausible-but-wrong chain, no hand-authoring required.

import random

def make_invalid_chain(correct_chain, objects):
    """Shuffle the bridging objects (entities/numbers) in a correct
    rationale to produce a plausible but incoherent wrong chain."""
    shuffled = objects[:]
    random.shuffle(shuffled)
    invalid = correct_chain
    for old, new in zip(objects, shuffled):
        invalid = invalid.replace(old, f"<<{new}>>", 1)
    return invalid.replace("<<", "").replace(">>", "")

# objects = the numbers/entities the reasoning hinges on
make_invalid_chain(
    "2 bolts of blue, half of 2 is 1 white, 2 + 1 = 3",
    objects=["2", "1", "3"],
)

Configuration

Implementation workflow

1. Get a CoT baseline. Build normal few-shot CoT and measure it on a held-out set. You need this number to prove contrastive CoT actually helps.
2. Author or generate wrong chains. For each demonstration, shuffle the bridging objects to create an incoherent chain. Sanity-check that it's plausible, not gibberish.
3. Assemble the prompt. Lay out each example as question, correct explanation, wrong explanation, then end with the target question asking for the correct explanation.
4. Run greedy first. Evaluate single-pass on the same held-out set and compare to the baseline.
5. Add self-consistency if needed. If you want more, sample several chains and majority-vote.

def contrastive_prompt(demos, question):
    blocks = []
    for d in demos:
        blocks.append(
            f"Question: {d['q']}\n"
            f"Correct explanation: {d['correct']}\n"
            f"Wrong explanation: {d['wrong']}"
        )
    blocks.append(f"Question: {question}\nCorrect explanation:")
    return "\n\n".join(blocks)

resp = client.chat.completions.create(
    model="gpt-3.5-turbo",
    temperature=0,
    messages=[{"role": "user",
               "content": contrastive_prompt(demos, target_q)}],
)

Do and don't

• Do keep the wrong chain the same length and style as the correct one, so the contrast is in the reasoning, not the formatting.
• Do corrupt the bridging objects specifically; that targets the failure mode CoT struggles with.
• Don't make the wrong chain obviously absurd. A cartoonish error is easy to ignore and teaches nothing.
• Don't forget the labels. Without a clear "wrong explanation" marker the model may imitate the bad chain.
• Don't drown the prompt in demonstrations; two chains per example eats context fast.

Debugging

• No improvement over CoT → your wrong chains may be too absurd to be instructive, or the task wasn't reasoning-bound to begin with. Make the errors subtler and confirm CoT actually slips here.
• Model copies the wrong chain → strengthen the labelling, or move the wrong chain before the correct one so the prompt ends on the right pattern.
• Answers got more verbose or drifted → trim the wrong chains; over-long negatives dilute the demonstration.
• Gains on dev, none in production → check your held-out set isn't leaking demonstration-like examples; re-measure on truly unseen data.

Proving it works

Treat it as an A/B test against your plain-CoT baseline on a held-out set. Use the same examples, same decoding, and change only whether the wrong chains are present. Accuracy (exact-match on the final answer) is the metric for these benchmarks.

def accuracy(samples, predict):
    correct = sum(predict(s["q"]) == s["gold"] for s in samples)
    return correct / len(samples)

cot = accuracy(test, cot_predict)
contrastive = accuracy(test, contrastive_predict)
print(f"CoT: {cot:.1%}   Contrastive: {contrastive:.1%}")

If the lift is marginal, try self-consistency on both arms before concluding, and vary the error type (incoherent objects usually wins).

Limitations

• It needs a reasoning task. On single-step classification or extraction, where CoT itself barely helps, the contrast adds cost without much benefit.
• It assumes a capable model. The technique was shown on GPT-3.5-Turbo-class models; a model that can't sustain a CoT chain won't gain from seeing a wrong one.
• Token overhead is real. Two chains per demonstration roughly doubles the rationale tokens, raising cost and latency.
• Wrong-chain quality is fiddly. The contrast only teaches if the error is plausible; absurd negatives are wasted, and crafting good ones by hand is work (hence the automatic shuffler).
• Diminishing headroom. As base CoT accuracy rises on stronger models, the room for contrastive gains shrinks.

Advanced techniques

Stack with self-consistency. This is the biggest lever. Sampling multiple contrastive chains and majority-voting drove the paper's largest numbers, into the high 80s on GSM8K and ASDiv.

Vary the error type per task. Incoherent objects is the default, but for tasks where the logical order of steps is the usual failure, an incoherent-language corruption (scrambled step order) may contrast more sharply.

Mix in multiple wrong chains carefully. More than one negative per demonstration is possible but spends context fast; weigh it against just adding a cleaner single contrast or more demonstrations.

Risks and ethics

The wrong chains are illustrative reasoning errors, not harmful content, so the safety surface is mostly about correctness rather than abuse. Standard prompt-injection hygiene on any user-supplied question still applies; contrastive CoT changes the demonstrations, not your input-validation needs.

Ecosystem

Contrastive CoT is a member of the chain-of-thought family and composes with most of its siblings.

Hybrids. The natural combination is contrastive CoT plus self-consistency, which the paper reports as its strongest configuration. Auto-CoT can supply the correct chains that the shuffler then corrupts, automating both halves.

Transitions. Coming from plain few-shot CoT, you keep your existing demonstrations and add a wrong chain to each, then re-measure. Going further, layer self-consistency, or move to retrieval if the residual errors turn out to be knowledge gaps rather than reasoning slips.

Future directions

The open questions are around automating high-quality negatives. The paper's object-shuffling method is simple and effective, but richer error taxonomies, model-specific failure mining, and applying the contrast to longer agentic reasoning are natural extensions. As base models grow more reliable at CoT, the interesting frontier shifts to harder, longer-horizon tasks where careless mid-chain errors still bite.

Summary

• Contrastive CoT pairs each correct reasoning demonstration with a labelled wrong one, so the model learns what to avoid, not just what to do.
• It targets the failure CoT can't see: prior work showed CoT helps even with invalid rationales, meaning the model never gets told which patterns are bad.
• Build wrong chains by shuffling the bridging objects (entities and numbers) of a correct chain; the paper automates exactly this.
• On GPT-3.5-Turbo it improved all seven tested benchmarks, e.g. GSM8K 69.2% to 79.0% and Bamboogle 40.8% to 56.8%, with self-consistency reaching 86.2% on GSM8K.
• Reach for it on multi-step reasoning where careless slips dominate; skip it on single-step tasks or when the bottleneck is missing knowledge.
• The main costs are token overhead (two chains per demonstration) and the effort of crafting plausible, not absurd, wrong chains.