Mixture of reasoning experts (MoRE): a complete guide

One prompt can't be good at everything. A chain-of-thought prompt that nails math questions falls apart on facts it never memorized, and a retrieval prompt that aces trivia stumbles on arithmetic. MoRE's flip is to stop forcing one prompt to generalize: run the same model under several reasoning-specialized prompts, then train a small selector to pick the best answer per question — or abstain. On 12 QA datasets spanning four reasoning types, MoRE reached 57.6% macro-average accuracy, beating the best single expert (49.6%) by 8 points (Si et al., EMNLP 2023 Findings).

See it work

Take one question that needs outside knowledge:

Question: In what year did the composer of "The Blue Danube" die?

Factual expert (retrieval):   1899   ✅  (Wikipedia passage on Johann Strauss II)
Multihop expert (CoT):        1849   ❌  (confuses him with Johann Strauss I)
Math expert (CoT):            1825   ❌  (no relevant reasoning to apply)
Commonsense expert (facts):   1900   ❌  (plausible guess, not grounded)

Answer selector → picks the Factual expert → 1899

No single prompt is right across all question types. The factual prompt wins here because retrieval grounds it. On a GSM8K word problem the math expert would win instead. The point of MoRE is that you don't have to know the question's reasoning type in advance — the selector learns to route after the fact.

The mental model

Think of a hospital front desk. A patient walks in; the receptionist doesn't diagnose anything. They send the case to the cardiologist, the neurologist, and the GP, then read the room — when several specialists converge on the same call, you trust it; when they scatter, you order more tests or refer out.

MoRE turns one model into a panel of specialists and trains a clerk to read their agreement.

The specialists are all the same backbone model wearing different prompts. The clerk is a lightweight classifier whose smartest feature is how much the experts agree.

How it works

1. Specialize by prompt, not by training. One backbone model (Codex in the paper) is turned into four experts purely through prompting:
   • Factual — retrieval-augmented prompting. Pull the top-10 Wikipedia passages with Contriever and prepend them before the question.
   • Multihop — chain-of-thought prompting with hand-written rationales after each demonstration, to elicit multi-step reasoning.
   • Math — chain-of-thought too, but with worked, GSM8K-style explanations after the demos.
   • Commonsense — generated-knowledge prompting. Have the model generate ~10 relevant facts first, then prepend them.
2. Run every expert on every question. You don't classify the question up front. Each expert produces an answer and a confidence signal.
3. Featurize the candidates. For each candidate, build features: the expert's identity, question traits (question word, length, whether it contains numbers), answer traits (its probability, length, token overlap with the question/context), and — the key ingredient — inter-expert agreement (how many experts produced the same answer, token overlap across answers).
4. Select or abstain. A small classifier scores the candidates and returns the best one. If confidence is low, MoRE abstains instead of guessing.

Why it works

Ranked roughly by how much each lever moves the result:

Where it shines

MoRE is built for mixed QA workloads where you can't predict which reasoning type the next question needs. Across the four families it covers:

• Factual QA — Natural Questions, TriviaQA, SQuAD.
• Multihop QA — HotpotQA, BeerQA, MuSiQue.
• Mathematical QA — GSM8K, SVAMP, MultiArith.
• Commonsense QA — CommonsenseQA, CSQA2.0, QASC.

On the full 12-dataset suite, the random-forest selector hit 57.6% macro-average accuracy versus 49.6% for the best single expert — an 8-point gain — while a plain majority vote managed only 43.6% and a max-probability baseline 49.7%. The oracle that always picks the right expert tops out at 69.9%, so there's headroom but the selector closes a large share of the gap.

The second payoff is selective QA: knowing when to abstain. Because agreement is interpretable, MoRE's calibrator decides when to trust an answer better than confidence-only baselines (risk-coverage AUC, where lower is better, improved once agreement features were added).

When to use it (and when not)

Reach for MoRE when:

• Your traffic mixes reasoning types and you can't route up front.
• Selective answering matters — you'd rather abstain than be confidently wrong.
• Interpretability matters; you want to show users why an answer was chosen.

Skip it when:

• All your questions are one type — just use that type's specialized prompt.
• Latency or cost is tight; you can't afford several expert calls plus retrieval per question.
• You have no labeled data to train the selector (the few-shot selector helps, but a trained one is stronger).

Model fit. The experts lean on a capable instruction-following backbone — the paper used Codex. Weak models won't produce the per-reasoning-type strength the selector depends on.

Escalation. If the selector's accuracy stalls well below the oracle ceiling, add features (or experts) before swapping the backbone. If you only ever see one reasoning type, drop MoRE for that single expert.

Structure and components

A MoRE system has three parts:

1. Experts — a set of reasoning-specialized prompts over one backbone. Required: at least two experts with genuinely different methods.
2. Feature extractor — turns each expert's output into a feature vector, including the cross-expert agreement signals.
3. Selector — a classifier that picks the winning candidate or abstains.

Core mechanism

The whole loop in pseudocode:

def more_answer(question, experts, selector, threshold):
    candidates = []
    for expert in experts:                 # factual, multihop, math, commonsense
        ans, conf = expert.run(question)   # same backbone, different prompt
        candidates.append({"expert": expert.name, "answer": ans, "conf": conf})

    feats = build_features(question, candidates)   # incl. inter-expert agreement
    best_idx, score = selector.predict(feats)

    if score < threshold:
        return ABSTAIN
    return candidates[best_idx]["answer"]

The agreement features are the part you can't skip:

def agreement_features(candidates):
    answers = [c["answer"] for c in candidates]
    return {
        "max_vote": max(answers.count(a) for a in answers),   # how many experts agree
        "n_distinct": len(set(answers)),                       # spread of answers
        "mean_overlap": mean_pairwise_token_overlap(answers),  # soft agreement
    }

Configuration

Implementation workflow

1. Pick reasoning types that match your traffic; write one specialized prompt per type.
2. Wire dependencies — retrieval for the factual expert, a knowledge-generation step for commonsense.
3. Run all experts over a labeled dev set and log answers plus confidences.
4. Build features, including the agreement signals, and label each example by which expert (if any) was correct.
5. Train the selector (random forest is a fine, interpretable default).
6. Calibrate abstention — set the threshold from the coverage/accuracy trade-off you want.
7. Evaluate against the best single expert and majority vote; check you're closing the gap to the oracle.

Do and don't

• Do keep experts methodologically diverse — retrieval, CoT, generated knowledge fail differently.
• Do include agreement features; they're the single biggest selection lever.
• Don't route by guessing the question type up front — let every expert run, then select.
• Don't ship without an abstention path if wrong answers are costly.

Debugging

Testing and how to prove it

Report macro-average accuracy across your reasoning types, not micro — micro hides weakness on rarer types. Always show three reference points: the best single expert, majority vote, and the oracle upper bound, so the selector's contribution is visible. For selective QA, plot the risk-coverage curve and report AUC (lower is better) plus coverage at a fixed accuracy target.

def macro_accuracy(results_by_dataset):
    per_dataset = [acc(r) for r in results_by_dataset]
    return sum(per_dataset) / len(per_dataset)   # equal weight per dataset

Limitations

• One backbone tested. Results are on Codex; behavior on other LLMs, especially open-source ones, isn't established.
• Four reasoning types only. Many question types are uncovered — multiple-answer, ambiguous, and false-presupposition questions among them.
• QA-only scope. MoRE was studied on question answering, not general-purpose generation.
• Selector needs data. The trained selector wants labeled examples; the few-shot fallback is weaker.
• Ceiling gap. The oracle hits 69.9%, so a meaningful share of correct expert answers still goes unselected.

Advanced and ecosystem

You can extend the expert roster to new reasoning types, or replace the random-forest selector with a few-shot in-context selector when you lack training labels. MoRE composes naturally with its building blocks: each expert is itself a known technique — RAG, chain-of-thought, generated-knowledge prompting — and MoRE is the meta-layer that ensembles and arbitrates them. Where self-consistency samples one prompt many times, MoRE samples many methods once and learns which to trust.

Future directions

Open threads from the paper: verify MoRE on more (and open-source) backbones, expand beyond four reasoning types to messier question forms, and push the framework past QA toward general-purpose generation. The deeper bet is that interpretable, agreement-based selection — not just bigger models — is how systems learn when to trust themselves.

Summary

• One prompt can't generalize across reasoning types; MoRE runs several reasoning-specialized prompts over one backbone instead.
• The four experts — factual (retrieval), multihop (CoT), math (CoT), commonsense (generated knowledge) — all run on every question.
• A trained selector picks the best answer or abstains, using inter-expert agreement as its strongest feature.
• It reached 57.6% macro-average accuracy on 12 QA datasets, beating the best single expert (49.6%) by 8 points (Si et al., EMNLP 2023 Findings).
• It improves selective QA and, by showing its work, helps humans decide when to trust it (67.5% vs 57.0%).
• Reach for it on mixed QA workloads where trust and abstention matter; skip it when traffic is one reasoning type or cost is tight.