Multimodal chain-of-thought (CoT): a complete guide

Text-only chain-of-thought reasons beautifully about words, but it can't see. Multimodal CoT fixes that: it folds the image into the reasoning chain so each step is grounded in what's actually on the screen, not a guess. In the founding work, a model under 1 billion parameters using this approach beat GPT-3.5 by 16 points on ScienceQA (75.17% to 91.68%) and edged past the human benchmark of 88.40% (Zhang et al., 2023, Multimodal Chain-of-Thought Reasoning in Language Models).

See it work

Ask a vision model a multi-step visual question with a plain prompt, and it tends to leap straight to an answer that ignores half the picture.

Prompt (direct):
[image: a beaker on a hot plate, thermometer reads 5°C, ice cubes floating]
Q: Will the ice melt faster if we raise the hot plate setting? Answer yes/no.

Output:
No.

Now ask it to reason over the image first, step by step. The chain forces it to read the thermometer, notice the heat source, and connect them before committing.

Prompt (multimodal CoT):
[same image]
Q: Will the ice melt faster if we raise the hot plate setting?
Let's look at the image and reason step by step before answering.

Output:
1. The beaker sits on a hot plate, which is a heat source.
2. The thermometer reads 5°C and ice cubes are still solid.
3. Adding more heat raises the water temperature.
4. Warmer water transfers heat to the ice faster.
Answer: Yes — raising the setting speeds up melting.

Same model, same image. The only change is that reasoning now runs through the visual evidence instead of around it.

The mental model

Imagine studying a textbook with every figure and diagram torn out. You could still read the words, but any question that hinges on a chart would be a coin flip. That's text-only CoT on a visual task. Multimodal CoT staples the figures back in.

Multimodal CoT is chain-of-thought that reads the picture: it interleaves visual evidence with verbal reasoning so each step is anchored to what the model can actually see.

How it works

The original paper found a trap: when a small model writes a reasoning chain from text alone, it hallucinates — it invents details the image would have contradicted. So Multimodal CoT splits the job into two stages and feeds vision into both.

1. Encode both modalities. The text (question, context, answer options) goes through a language encoder; the image goes through a vision encoder (the paper used DETR features, and also tested ViT and CLIP).
2. Fuse them. A gated fusion layer combines text and vision representations so visual detail is preserved instead of being flattened into a caption.
3. Stage 1 — generate the rationale. Given question, context, options, and image, the model writes the reasoning chain (the "QCM to R" step).
4. Stage 2 — infer the answer. The generated rationale is appended to the original input, and the model — again seeing the image — predicts the final answer (the "QCMR to A" step).

Both stages share the same architecture but differ in input and output. The split matters: it stops a shaky rationale from short-circuiting straight to a wrong answer, and it lets answer inference lean on a vision-grounded chain.

Why it works

Where it shines

Multimodal CoT pays off whenever the answer depends on reading the image and reasoning over it:

• Science and diagram QA. On ScienceQA — a benchmark of multimodal science questions — the two-stage model reached 91.68%, above the 88.40% human average and 16 points over the GPT-3.5 baseline of 75.17%.
• Visual question answering. The authors also evaluated on A-OKVQA, a VQA benchmark that demands outside knowledge plus visual grounding.
• Modern vision LLM prompting. The same idea drives "reason step by step about this image" prompts on today's large vision-language models for chart reading, document understanding, screenshots, and math-with-figures.

The headline finding is the one to remember: explicit visual grounding mitigates the hallucinated rationales that text-only chains produce, and that's what closes the gap.

When to use it (and when not)

Reach for it when:

• The question can't be answered from text alone — it needs a chart, diagram, photo, or screenshot.
• The task is multi-step: read the image, combine it with knowledge, then decide.
• Text-only CoT is hallucinating details the image would settle.

Skip it when:

• The image is decorative and the answer lives entirely in the text — plain CoT is cheaper.
• The task is a single-glance lookup ("what color is the car?") — a direct prompt is enough.
• You're on a text-only model with no way to encode the image.

Model fit. The original recipe fine-tunes a small encoder-decoder (T5/UnifiedQA, 223M base or 738M large) with a vision encoder and gated fusion — strong when you can train and want a tiny, cheap model. If you'd rather not train, a capable vision-language model prompted to reason step by step gives you the same behavior with no fine-tuning.

Escalation. If a one-pass "reason then answer" prompt still hallucinates, split it into two explicit calls (rationale, then answer) so the chain is grounded before the model commits.

Structure and components

A Multimodal CoT setup has four moving parts:

• A vision encoder that turns the image into feature vectors (DETR, ViT, or CLIP in the paper).
• A language model that reads the question and writes the rationale and answer.
• A fusion mechanism — gated fusion — that combines the two modalities while keeping visual detail.
• A two-stage control flow that generates the rationale first, then conditions the answer on it.

The prompt-side template for the model-agnostic, prompted version is simple:

[IMAGE]
Question: <question>
Options: <A> <B> <C> <D>   # if multiple choice
Instruction: Look at the image and reason step by step.
First write your reasoning, then state the final answer.

The mechanism in code

Here's the two-stage flow as you'd run it against a vision-capable model. Stage 1 produces the rationale from the image and question; stage 2 feeds that rationale back, with the image, to get the grounded answer.

import base64
from anthropic import Anthropic

client = Anthropic()

def encode(path: str) -> dict:
    data = base64.standard_b64encode(open(path, "rb").read()).decode()
    return {"type": "image",
            "source": {"type": "base64", "media_type": "image/png", "data": data}}

def ask(image_block, text: str) -> str:
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        messages=[{"role": "user", "content": [image_block, {"type": "text", "text": text}]}],
    )
    return "".join(b.text for b in msg.content if b.type == "text")

img = encode("diagram.png")
question = "Will the ice melt faster if we raise the hot plate setting?"

# Stage 1: generate a vision-grounded rationale
rationale = ask(img, f"{question}\nReason step by step about the image. Reasoning only, no answer.")

# Stage 2: infer the answer, conditioned on the rationale + image
answer = ask(img, f"{question}\nReasoning: {rationale}\nGiven this, state the final answer.")
print(answer)

A single-call version just asks for reasoning and the answer in one prompt; the two-call split is the escalation when a one-pass chain still hallucinates.

Configuration

Implementation workflow

1. Confirm the task actually needs the image — otherwise use text CoT.
2. Pick the path: fine-tune the two-stage recipe, or prompt a vision LLM.
3. For prompting, start with a single "reason step by step about the image" call.
4. Check for hallucinated rationales — claims the image contradicts.
5. If they appear, split into two stages so the answer is conditioned on a grounded chain.
6. Tune image resolution against cost; cache encodings you reuse.

Do and don't

• Do tell the model to look at the image before reasoning — ordering matters.
• Do keep the image present in both stages; dropping it in stage 2 reintroduces hallucination.
• Do prefer raw image features or the actual image over a text caption when detail matters.
• Don't let the chain run unbounded — cap output so it doesn't ramble past the answer.
• Don't pay for image tokens on tasks the text already answers.

Debugging

• Answer ignores the image. The model is reasoning from text priors — make the instruction explicitly reference the image and put the image first.
• Rationale invents details. Classic text-only-chain hallucination; ground it by ensuring vision is fused into stage 1, or split into two stages.
• Right reasoning, wrong answer. The answer step isn't using the rationale — feed the rationale back explicitly in stage 2.
• Costs spiking. Image resolution or chain length is too high; downsample and cap output tokens.

Limitations

• Doubled cost and latency. Image tokens plus a reasoning chain, and the two-stage variant runs the model twice.
• Modality alignment is hard. Fusing vision and language representations well is the core engineering challenge; weak fusion drops detail.
• Fine-grained perception still fails. If the vision encoder can't read the small print, no amount of reasoning recovers it.
• Grounding is a mitigation, not a cure. Vision features cut hallucinated rationales sharply but don't eliminate them.

Advanced techniques

• Interleaved reasoning. Instead of one block of text after the image, alternate between looking and reasoning — useful for images with several regions to inspect in turn.
• Self-verification. After the answer, ask the model to re-check each reasoning step against the image and flag any claim it can't see.
• Region prompting. Crop or point to the relevant region so the chain reasons over the right part of a dense image.

Ecosystem

The cleanest way to adopt it today is to take a working text CoT prompt, add the image, and instruct the model to reason over what it sees. When that one-pass chain hallucinates, graduate to the two-stage split.

Summary

• Multimodal CoT extends chain-of-thought so reasoning is grounded in the image, not just the text.
• The founding paper (Zhang et al., 2023, published in TMLR 2024) used a two-stage design: generate a vision-grounded rationale, then infer the answer from it.
• A sub-1B model hit 91.68% on ScienceQA — 16 points over GPT-3.5 (75.17%) and above the human 88.40%.
• Vision features corrected 60.7% of the hallucinated rationales that text-only chains produce.
• Reach for it on multi-step visual tasks; skip it when the text alone holds the answer.
• Today you can prompt any capable vision LLM to reason step by step over an image, and split into two stages when a one-pass chain drifts.