Skeleton-of-thought: a complete guide

Large language models are slow for one structural reason: they write one token at a time, in order, even when the answer is really a list of independent points. Skeleton-of-Thought (SoT) flips that. It asks the model for a bare outline first, then expands every bullet at the same time, in parallel. On the original benchmark it hit up to a 2.39x speed-up while often improving answer quality (Ning et al., 2023, ICLR 2024).

See it work

Ask a model a list-shaped question the normal way and you wait for the whole thing to stream out, sentence by sentence, top to bottom.

Prompt: What are the main benefits of remote work?

Output (one sequential pass, ~22s):
Remote work offers several benefits. First, it eliminates the daily
commute, which saves time and money... [continues writing every
sentence in order, start to finish]

SoT splits the same request into two stages. First, a skeleton:

Stage 1 — Skeleton prompt:
You are an organizer. Provide a skeleton for the answer — a list of
short point headers, 3-5 words each.

Skeleton output:
1. Saves commute time
2. Lower living costs
3. Flexible schedule
4. Wider talent pool
5. Fewer office distractions

Then five expansion calls fire at once, each fleshing out one header in 1-2 sentences:

Stage 2 — Point-expanding (5 parallel calls):
1. Saves commute time: Skipping the daily commute reclaims 1-2 hours
   a day and cuts transport spending.
2. Lower living costs: Workers can live in cheaper areas instead of
   pricey job-center cities.
... (points 3-5 returned concurrently, then merged)

The wall-clock time is now roughly the skeleton call plus the longest single expansion, not the sum of all five. Same answer shape, a fraction of the wait.

The mental model

Think about how a person writes under time pressure. You don't compose a five-paragraph answer word by word from the top. You jot a quick outline, then flesh out each bullet, often in any order. SoT hands the model that same workflow.

SoT trades sequential decoding for an outline plus a batch of short, independent fill-in tasks. The skeleton is the cheap serial part; the expansions are the embarrassingly parallel part.

The key realization from the paper: the latency bottleneck isn't compute, it's the sequential dependency of autoregressive decoding. Break the answer into pieces that don't depend on each other, and you can decode them simultaneously.

How it works

1. Skeleton stage. One prompt asks the model for a numbered list of point headers — concise, 3-5 words each, typically 3-10 points total. This call is short, so it's fast.
2. Parse the skeleton. Pull the N headers out of the response with a simple regex or split.
3. Point-expanding stage. Issue N expansion requests, each carrying the question plus the full skeleton plus "continue only point i, in 1-2 sentences." These run together.
4. Run them in parallel. For API models, that's N concurrent API calls. For models you host, it's a single batched decode where the N prompts share one forward pass.
5. Merge. Concatenate the expanded points back in skeleton order into the final answer.

The parallelism is the whole point. Sequential generation pays for every token of the final answer end to end. SoT pays for the skeleton plus only the slowest branch.

Why it works

Where it shines

SoT helps most when the answer is naturally a list of parallel, self-contained items. In the paper's evaluation across 12 models, it earned a speed-up above 2x on 8 of them, topping out at 2.39x, with per-category speed-ups landing between 1.89x and 2.33x on the categories it suits.

By question type, SoT did well on generic, common-sense, knowledge, roleplay, and counterfactual questions — the kind where you can enumerate independent angles. Good fits in practice:

• "List the pros and cons of X" / "What are the main causes of Y?"
• Brainstorming, feature lists, checklists, multi-part explanations.
• Roleplay or open-ended advice that decomposes into separate themes.
• Knowledge dumps where each point is a distinct fact or angle.

On quality, the GPT-4 judge in FastChat scored SoT answers a 45.8% win rate against normal generation, and across metrics SoT was "not worse" than the baseline in roughly 60% of cases. The LLMZoo evaluation found SoT actually improved diversity and relevance, while coming out slightly behind on coherence and immersion — exactly what you'd expect when an outline boosts breadth but parallel writing loses some connective flow.

When to use it (and when not)

Reach for SoT when:

• Latency is a real user-facing cost and the answer is list-shaped.
• The points are genuinely independent of each other.
• You can run concurrent API calls or batched decoding.

Skip it when:

• The task needs sequential reasoning: math, multi-step coding, logical proofs. SoT did relatively poorly here because later steps depend on earlier ones — exactly what parallel expansion breaks.
• The answer isn't a list (a single flowing narrative, a short factual reply).
• The paper also saw weaker results on writing and fermi (estimation) questions, where coherence across the whole piece matters more than breadth.

Model fit: the paper tested 9 open-source models (LLaMA2-Chat-7B/13B, OpenChat-13B, Vicuna-7B/13B/33B V1.3, StableVicuna-13B, UltraLM-13B, Vicuna-7B V1.1) and 3 API models (Claude, ChatGPT-3.5, GPT-4). Speed-ups appeared across the board, so SoT isn't tied to one model family — it needs a model strong enough to follow the two-stage instructions and an inference setup that can actually run the expansions concurrently.

The two-stage prompts

SoT lives entirely in two prompt templates — no fine-tuning, no special decoding.

The skeleton prompt constrains the outline so the points stay short and parallelizable:

You're an organizer responsible for an answer's outline. Provide the
skeleton for the answer to the question below. Give a list of points.
Each point should be very short — only 3-5 words. The skeleton should
have 3-10 points. Don't elaborate; just the headers.

Question: {question}
Skeleton:
1.

The point-expanding prompt fleshes out exactly one header and is told to stop there, which keeps every branch short:

You're continuing an answer. Here is the question and the skeleton.
Continue ONLY point {i}. Write it very shortly, in 1-2 sentences.
Do not continue to any other point.

Question: {question}
Skeleton: {skeleton}
{i}. {header}:

The original work used 2-shot demonstrations in these templates for most models (GPT-4 ran zero-shot), to lock in the format.

A minimal implementation

The orchestration is small. Get the skeleton, split it, fan out the expansions concurrently, then stitch.

import asyncio, re

async def skeleton_of_thought(client, question):
    # Stage 1: skeleton
    skeleton = await client.complete(skeleton_prompt(question))
    points = re.findall(r"^\s*\d+\.\s*(.+)$", skeleton, re.MULTILINE)

    # Stage 2: expand every point concurrently
    tasks = [
        client.complete(point_prompt(question, skeleton, i + 1, p))
        for i, p in enumerate(points)
    ]
    expansions = await asyncio.gather(*tasks)

    # Merge in skeleton order
    return "\n".join(f"{i+1}. {e.strip()}" for i, e in enumerate(expansions))

For self-hosted models, swap the concurrent calls for a single batched decode: pass all N expansion prompts as one batch so they share GPU forward passes. That's where open-source models get their gains, since you're not paying per-call network overhead.

SoT with router (SoT-R)

Plain SoT has one obvious flaw: it forces an outline onto every question, including the math and coding ones it's bad at. SoT-R fixes that by adding a gatekeeper that decides, per question, whether to apply SoT or fall back to normal sequential generation.

The paper offers two routers:

• Prompting router. Ask GPT-4 to classify whether the question's answer is a set of independent points (use SoT) or needs dependent, sequential reasoning (use normal generation).
• Trained router. A small RoBERTa classifier (about 120M parameters) fine-tuned on the LIMA dataset (1,030 Q&A pairs) with a Tversky loss (α=0.7, β=0.3). Training took about 2 minutes on an A100.

SoT-R gives up some of the headline speed-up — the router adds overhead and routes hard questions away from SoT — but it recovers quality on the categories where plain SoT stumbled. The trained router matched plain routing on Vicuna-80 and beat it on the harder WizardLM set, and both routers could outperform a human-written routing rule on some categories.

Limitations

• No help for sequential tasks. Anything where step k depends on step k-1 (math, multi-step code, proofs) breaks under parallel expansion. This is fundamental, not a tuning issue.
• Points can't reference each other. Because branches expand blind to one another, you can get repetition across points or a loss of cross-point coherence. This is why immersion and coherence scored slightly lower.
• Higher token cost. More calls and longer answers mean more tokens billed — you optimize latency, not spend.
• Format dependence. A weak skeleton (too few points, vague headers, or a refusal to produce a list) drags the whole pipeline down. Demonstrations help, but smaller models are shakier.
• Infra requirement. The speed-up only materializes if you can actually run expansions concurrently; a serial loop over the points gives you no benefit at all.

Where it fits in the ecosystem

SoT is a latency technique, which sets it apart from its cousins. Chain-of-Thought adds sequential reasoning steps to raise accuracy — the opposite goal, and structurally incompatible with parallel decoding. Self-consistency samples many full reasoning paths and votes, multiplying cost for quality. SoT instead decomposes a single answer into parallel chunks purely to cut the wait.

It composes cleanly with retrieval (expand each point against retrieved context) and slots into agent pipelines wherever a step produces a list-shaped output that a user is waiting on.

Summary

• The flip: outline first, then expand every point in parallel, instead of writing one token at a time start to finish.
• The payoff: up to 2.39x lower latency (above 2x on 8 of 12 models), often with equal or better quality on list-shaped questions.
• The cost: more API calls, longer answers, higher token spend — you buy speed, not savings.
• Where it wins: independent-point answers — knowledge, common-sense, brainstorming, roleplay.
• Where it fails: sequential reasoning — math, coding, proofs, tightly-woven narratives.
• The safety net: SoT-R routes each question to SoT or normal generation, so mixed traffic doesn't degrade.
• No training needed: it's two prompt templates plus concurrent (or batched) execution.