Self-ask: a complete guide

A model can often answer "When was Albert Einstein born?" and "Who was US president in 1879?" on their own, yet fumble "Who was president when Einstein was born?" That blind spot has a name: the compositionality gap — the fraction of multi-hop questions a model gets wrong even though it knows every sub-fact. Press et al. (2022) measured it at a stubborn ~40% across GPT-3 sizes, and introduced self-ask to shrink it: have the model explicitly ask and answer its own follow-up questions first. On their Bamboogle benchmark it lifted accuracy from 46.4% (chain of thought) to 57.6%, and to 60.0% once a search engine answered the follow-ups.

See it work

Ask a strong model a two-hop question cold, and it tends to pattern-match its way to a confident, wrong guess:

Prompt:
Who lived longer, Theodor Haecker or Harry Vaughan Watkins?

Output (direct):
Theodor Haecker.        ← guessed, never checked either lifespan

Now give it a scaffold that forces the intermediate steps out into the open:

Prompt (self-ask):
Question: Who lived longer, Theodor Haecker or Harry Vaughan Watkins?
Are follow up questions needed here:

Output:
Are follow up questions needed here: Yes.
Follow up: How old was Theodor Haecker when he died?
Intermediate answer: Theodor Haecker was 65 years old when he died.
Follow up: How old was Harry Vaughan Watkins when he died?
Intermediate answer: Harry Vaughan Watkins was 69 years old when he died.
So the final answer is: Harry Vaughan Watkins.

Same model, same facts. The only change is that the second prompt makes it decompose the question and answer each piece before composing the final answer. That's the whole idea.

The mental model

Think of a journalist handed a hard question. They don't blurt out an answer. They break it into smaller questions they can check — "How old was each person when they died?" — look each up, then combine the findings. Self-ask hands the model that same habit as a fixed template it fills in.

Self-ask turns one impossible leap into a short chain of easy hops, and writes every hop down.

How it works

Self-ask is a few-shot prompt. You show the model a handful of examples that all follow one rigid scaffold, then append your real question with the trailing cue Are follow up questions needed here:. The model continues the pattern: it decides whether it needs sub-questions, asks them one at a time, answers each, and finally emits a line starting So the final answer is:.

Step by step:

1. Prime with demos. Two to four worked examples teach the exact format: the Are follow up questions needed here: opener, Follow up: sub-questions, Intermediate answer: replies, and the closing So the final answer is:.
2. Append the new question plus the opener cue, then call the model.
3. Decompose. The model emits one follow-up sub-question.
4. Answer the sub-question — either from the model's own knowledge, or, in the search variant, from an external source.
5. Loop until no more follow-ups are needed.
6. Compose the final answer from the intermediate answers.

The scaffold does two jobs at once: it elicits reasoning (like chain of thought) and it carves that reasoning into clearly labelled, machine-parsable slots.

Why it works

The structure is the real edge over plain chain of thought. CoT reasons in free-form prose; self-ask reasons in named fields, which is exactly what lets you bolt a search engine onto the Follow up: lines.

Where it shines

Self-ask targets compositional, multi-hop questions — ones whose answer requires chaining two or more facts that were probably never seen together in training. Press et al. built two datasets to stress this:

• Compositional Celebrities — 8,600 auto-generated 2-hop questions across 17 categories (e.g. "the birth year of the performer of a given song").
• Bamboogle — 125 hand-written 2-hop questions, each crafted so that a plain search engine query returns the wrong answer, but both supporting facts exist in Wikipedia.

On Bamboogle, the ladder is clear: direct prompting 17.6%, chain of thought 46.4%, self-ask 57.6%, and self-ask with a search engine 60.0%. The gains are largest on diverse, heterogeneous questions like Bamboogle; on the more uniform Compositional Celebrities, chain of thought lands within about 1% of self-ask, because once you've memorised one template you don't need the extra structure.

When to use it (and when not)

Reach for self-ask when:

• The question is genuinely multi-hop — the answer depends on composing two or more facts.
• You want to plug a retrieval step or search engine into the reasoning.
• You need the intermediate steps in a structured, inspectable form.

Skip it when:

• The task is single-hop or pure classification — the scaffold is pure overhead.
• The reasoning is mathematical or procedural rather than fact-lookup; plain chain of thought or program-aided prompting fits better.
• Latency and cost are tight and the question is easy.

Model fit. Self-ask leans on instruction-following and in-context pattern matching, so it pays off most on capable models. The original work used GPT-3 davinci-002 and InstructGPT variants up to 175B parameters; weaker models drift from the scaffold and stop emitting clean Follow up: lines.

Alternatives at a glance:

The scaffold and its components

Every self-ask prompt is built from five fixed markers. Keep them verbatim — the model keys off the exact strings.

Question: <the user question>
Are follow up questions needed here: <Yes or No>
Follow up: <sub-question>
Intermediate answer: <answer to the sub-question>
Follow up: <next sub-question>
Intermediate answer: <answer>
So the final answer is: <composed answer>

The required pieces: a leading Question:, the Are follow up questions needed here: decision line, zero-or-more Follow up: / Intermediate answer: pairs, and exactly one So the final answer is: closer. In rare cases the model answers No to the decision line and jumps straight to the final answer — that's expected for questions it can handle in one hop.

A minimal implementation

The structure makes parsing trivial: split on the marker strings, stop generation at the next Intermediate answer: so you can supply the answer, then feed it back. Here's the search variant in pseudocode.

FEWSHOT = open("self_ask_demos.txt").read()  # 2-4 worked examples

def self_ask(question, search):
    prompt = FEWSHOT + f"\nQuestion: {question}\nAre follow up questions needed here:"
    while True:
        # Stop as soon as the model proposes a follow-up to answer it ourselves.
        text = llm(prompt, stop=["Intermediate answer:"], temperature=0)
        prompt += text
        if "Follow up:" in last_line(text):
            sub_q = text.split("Follow up:")[-1].strip()
            answer = search(sub_q)                 # external lookup
            prompt += f" Intermediate answer: {answer}\n"
        else:
            break  # model emitted "So the final answer is:"
    return prompt.split("So the final answer is:")[-1].strip()

Drop the search call and let the model answer its own follow-ups, and you have plain self-ask without retrieval.

Configuration

Do and don't

• Do keep the marker strings byte-for-byte identical between demos and runtime.
• Do pick demos whose decomposition style matches your target questions.
• Don't mix question domains wildly in one demo set; it muddies the pattern.
• Don't forget a hop cap — a confused model can ask follow-ups forever.

Debugging

• No Follow up: lines appear → your demos are too few or the markers don't match; the model never learned the format.
• Wrong final answer despite correct intermediates → a composition failure; add a demo that models the exact combine step (e.g. comparing two numbers).
• Search returns junk → the follow-up sub-question is under-specified; demos with crisper sub-questions help the model phrase searchable queries.
• Endless follow-ups → enforce the max-hop cap and stop sequence.

Testing and proving it

Measure self-ask the way the paper did: exact-match accuracy on held-out multi-hop questions, compared against a direct-prompting baseline and a chain-of-thought baseline on the same questions. To see whether your wins come from reasoning or from retrieval, run plain self-ask and the search variant side by side. If search barely moves the needle, your model already knows the facts and the structure is doing the work; if it jumps, you were memory-bound.

Limitations

• It's a fact-composition tool, not a reasoner. Self-ask shines on multi-hop lookup; it does little for arithmetic-heavy or purely logical problems.
• The compositionality gap doesn't vanish. Self-ask narrows it but doesn't close it — composition still fails sometimes even when every sub-answer is right.
• Garbage sub-answers poison the result. A wrong Intermediate answer: (from the model or a bad search hit) propagates straight into the final answer.
• Format fragility. Weaker models drift off the scaffold, breaking your parser.
• Cost and latency. Multiple generations and search calls per question add up.

Ecosystem and related techniques

Self-ask is built into common tooling: LangChain ships a Self-Ask with Search agent that wires the scaffold to a search tool out of the box, and the original code and datasets (Compositional Celebrities, Bamboogle) live in the ofirpress/self-ask repo.

It sits between two neighbours. Chain of thought is its less-structured ancestor — self-ask is essentially CoT with named slots so you can intercept the reasoning. ReAct is its more agentic descendant — where self-ask only asks-and-answers fact follow-ups, ReAct interleaves free-form thoughts with arbitrary tool actions. A natural progression: start with chain of thought, switch to self-ask when you need structure or retrieval, and graduate to ReAct when one tool and one question type stop being enough.

Future directions

The durable contribution here is the interface, not just the accuracy bump: by giving reasoning a parseable shape, self-ask turned "let the model think" into "let the model think, and let your code step in between thoughts." That pattern — structured, interceptable intermediate steps — underpins much of today's retrieval-augmented and tool-using agent design. Open questions remain around dynamically deciding when a follow-up needs search versus memory, and around closing the residual compositionality gap that even self-ask leaves behind.

Summary

• The compositionality gap is the ~40% of multi-hop questions a model botches despite knowing every sub-fact (Press et al., 2022).
• Self-ask is a few-shot scaffold that makes the model ask and answer its own Follow up: questions before stating So the final answer is:.
• It beat chain of thought on Bamboogle (57.6% vs 46.4%), and its labelled slots let you plug in a search engine, reaching 60.0%.
• Reach for it on multi-hop fact composition, especially when you want retrieval or inspectable steps; skip it for single-hop, math, or latency-critical tasks.
• Keep the marker strings exact, run at temperature 0, cap the hops, and treat sub-answers as fallible.
• It's the structured middle ground between chain of thought and ReAct.