Max mutual information method: a complete guide

Few-shot accuracy on the same model and the same task can swing by up to 30 percentage points just from rewording the instruction, reordering examples, or relabeling the classes (Zhao et al., 2021). So which prompt template do you ship when you have no labels to test against? Max Mutual Information (MMI) answers that with a single flip: instead of measuring accuracy, measure how much the model's output depends on the input. Sorensen et al. (2022) showed this unsupervised signal recovers about 90% of the gap between an average template and the best one — without ever seeing a ground-truth label.

The intuition is that a good template makes the model do two things at once: produce different answers for different inputs (it actually reads the input instead of defaulting to one label), and produce confident answers for each input (it has a clear opinion). Mutual information measures exactly that — the information the output carries about the input. Rank your templates by it and pick the top one.

See it work

Say you're classifying movie-review sentiment and you can't decide between two phrasings. You have 50 unlabeled reviews. You run each review through both templates, read off the model's probability for "positive" versus "negative", and score the templates.

Template A — bare completion
{text}
Answer:

Template B — named-label instruction
Classify the sentiment of this review as positive or negative.
Review: {text}
Sentiment:

--- feed both 50 reviews, then look at what the model does (illustrative) ---

Template A:  predicts "positive" on 47/50 reviews, mostly low confidence
             -> outputs barely depend on the input -> low mutual information

Template B:  predicts a clear positive/negative split, high confidence each
             -> outputs track the input -> high mutual information

MI(A) = 0.06 nats      MI(B) = 0.55 nats   -> ship Template B

Template A collapsed onto the majority label — a classic prompting failure. MMI caught it with zero labels, because a template that always says "positive" carries no information about the input. You never looked at a single correct answer.

The mental model

Think of an interviewer screening candidates. A bad interviewer asks one vague question and gives everyone the same lukewarm rating — their scores tell you nothing about who's actually good. A good interviewer asks sharp questions, gets distinct reads on each person, and commits to a clear verdict. You can spot the good interviewer without knowing the "right" hire: just check whether their verdicts vary across candidates and whether each verdict is decisive.

A template is good when the model's answer changes with the input and stays confident for each input. That's mutual information, and you can measure it without labels.

How it works

MMI is a single-pass, pre-deployment selection procedure. You score a fixed pool of templates once, pick the winner, and use it for all later inference at no extra cost.

1. Build a template pool. Assemble two or more candidate templates. Each fills the input at a fixed position and ends on a completion cue ("Sentiment:", "Answer:") where the next token should be a label. Write them by hand, generate them with an LLM, or pull from prior runs.
2. Sample unlabeled inputs. Draw n real inputs from your deployment domain. No labels needed — the paper recommends scoring on the actual test inputs since that maximizes distributional fidelity.
3. Collect probabilities. For each template and input, get p(y | x, t) for every candidate label y. This needs token-level log-probabilities. Single-token labels are one logit lookup; multi-token labels ("positive" → "pos" + "itive") should sum log-probs across all tokens (the all-token approach, which beats the one-token approximation).
4. Compute MI per template from the probability matrix.
5. Select t* = argmax_t MI(t) and deploy it.

The scoring objective is Shannon mutual information between output Y and input X under template t:

I(Y ; X | t) = H(Y | t) - H(Y | X, t)

  H(Y | t)     marginal entropy: high when the model spreads predictions
               across classes; zero when it always predicts one label.

  H(Y | X, t)  average conditional entropy: low when the model is confident
               per input; high when it's confused regardless of input.

  t* = argmax_t [ H(Y | t) - H(Y | X, t) ]

A high score needs both high marginal entropy (diversity across inputs) and low conditional entropy (confidence per input). A template that always predicts "positive" has zero marginal entropy; one that's uniformly unsure has maximum conditional entropy. MMI rejects both pathologies, and empirically the templates that survive are the ones the model was trained to interpret correctly.

Why it works

The model's training calibrated it to answer confidently when an input clearly expresses a class under a familiar format. A template that triggers that calibrated behavior produces confident, input-dependent predictions — exactly what high MI rewards. Here's what actually moves the score, ranked:

The effect is stronger on larger, better-calibrated models. GPT-4-class and 70B+ models are genuinely more confident on inputs they get right and less confident on ones they miss, so MI tracks accuracy tightly. Smaller models are overconfident, so they can post high MI even on wrong predictions.

Where it shines

MMI fits any task whose outputs are a finite, enumerable set of string labels.

It does not apply to free-form output: summarization, translation, open-domain QA, code generation, regression, or structured prediction like parsing and coreference. There's no finite label vocabulary to compute MI over.

On the empirical side, Sorensen et al. (2022) evaluated across 8 datasets spanning 7 NLP tasks (sentiment, NLI, topic classification, and others) and closed 90% of the average-to-best-template gap on the largest models — dropping to roughly 50–70% on smaller models, since the MI-accuracy correlation firms up with scale. That beat MDL-based selection (20–40% gap closure, per Perez et al., 2021) and cross-validation baselines by wide margins.

Yang et al. (2024) then ran a unified evaluation across 13 datasets and 10 decoder models (1.3B to 66B parameters), measuring oracle-F1 recovery:

Domain evidence backs this up. A 2024 JMIR scoping review of 114 prompt-engineering studies in medicine (2022–2024) found prompt design was the dominant approach in 78 of them, on exactly the closed-vocabulary, label-scarce tasks MMI targets (clinical sense disambiguation, medication-attribute extraction, symptom classification). JAMIA 2024 work on clinical NER found best-performing prompts substantially beating naive ones on identical models. Beyond classification, MI-family methods generalize: PMI scoring of RAG document orderings gained 2–3 points on NQ-Open across LLaMA-2/3, Mistral, and MPT (arXiv:2411.07773), and an MI-style scorer with a learned evaluator (APS, arXiv:2404.02717) hit 81.49% on GSM8K and 100% on MultiArith — though learned evaluators outperform pure MI on multi-step reasoning, confirming MMI's home turf is classification.

When to use it (and when not)

Reach for MMI when: you have multiple template phrasings and no idea which is best; you have no ground-truth labels; you see high output variance across near-identical templates; you need a principled, reproducible ranking; your task is classification with a known label set and an API that exposes logprobs.

Skip it when: output is free-form; only one template exists; your input sample is tiny (fewer than 10 inputs makes marginal estimates unreliable); labels are continuous or ordinal; the API hides logprobs (standard Claude or Gemini chat); or per-request latency is so tight you can't afford instance-wise selection.

Model fit. The floor is any model with logprob access and non-trivial classification ability (GPT-3-class, or 7B+ instruction-tuned open models). Sweet spot is well-calibrated models (GPT-3.5, LLaMA-3 8B+, Mistral 7B+). Best is GPT-4-class or 70B+. Not suitable: models below 1B with poor calibration, or pure chat-only APIs without logprobs. Below 3B, calibration is poor and MI barely beats random; 7B–13B gives meaningful value; 70B+ is most reliable.

Escalate when: MI_AGL + CBM still isn't accurate enough → move to ProTeGi or OPRO with a small labeled set; no template scores high → the model may not support the task at this capability, so fine-tune or upgrade; scores swing across sample draws → raise n or question whether the input distribution is stable enough for any unsupervised method.

MMI is one option among template optimizers. Here's how it sits against the alternatives:

Prefer MMI when no labels exist (it dominates MDL), when labels are expensive (use the few you have for validation only), when you need human-readable prompts (RLPrompt produces gibberish), or when your template pool is fixed. Prefer the alternatives when you have labels to spare (ProTeGi, OPRO raise the accuracy ceiling), when you need to discover new phrasings (APE, OPRO), or when the task is generation (APE/OPRO with ROUGE, BLEU, BERTScore).

How MMI relates to its siblings

Yang et al. (2024) proved a unifying result: every probability-based prompt-selection method is a special case of MI. Global Entropy (GE) estimates only the marginal-entropy term and recovers ~72% of oracle F1. Minimum Description Length (MDL) estimates only the conditional-entropy term and recovers ~65%. MMI captures both. Domain-Conditional PMI (PMI_DC) from Holtzman et al. (2021) addresses surface-form competition — different phrasings of the same answer ("car" vs. "automobile") splitting probability mass — via log p(answer | question, domain) - log p(answer | domain). It's a per-prediction decoding fix, complementary to MMI's per-template selection.

Contextual Calibration (CC) from Zhao et al. (2021) corrects label bias by passing a content-free input ("N/A") through the template and subtracting the resulting prior. On SST-2 it lifted GPT-3 1-shot accuracy from 67.3% to 79.1%. But Zhao's deeper contribution was diagnosing three biases that make prompting fragile: majority-label bias (predicting whichever label dominates the in-context examples), recency bias (over-predicting labels near the end of the prompt), and common-token bias (over-predicting labels whose surface forms are frequent in pretraining). MMI is structurally immune to the first two — a template that always predicts the majority label has zero mutual information.

Structure and components

MMI isn't a template — it's a procedure for ranking templates. Its required parts:

1. Template pool {t₁, ..., tₖ} — at least two candidates, all sharing the same label vocabulary so cross-template MI is comparable.
2. Label vocabulary Y — the finite class set, mapped to scoreable token strings. "Positive"/"Negative" vs. "positive"/"negative" vs. "pos"/"neg" all yield different estimates.
3. Unlabeled input sample — real domain inputs. Minimum ~20–50; use 100+ for production.
4. Logprob access — the endpoint must return token-level log-probabilities for label tokens (OpenAI Completions, Hugging Face, most open-source providers; not Anthropic or Google chat APIs).
5. MI computation — the scoring function for I(Y;X|t).

Optional but valuable: CBM calibration (recommended), all-token aggregation for multi-token labels, instance-wise selection (more accurate, adds latency), and a diversity filter to keep the pool from collapsing into near-duplicates.

Design templates so the label tokens land right after a clear completion cue, where the model's next-token prediction is the class decision. Three patterns cover most needs:

Standard — instructional framing (recommended default)
Classify the sentiment of the following text as positive or negative.
Text: {text}
Sentiment:

Few-shot — calibrates label meaning, lowers conditional entropy
Classify the sentiment.
Text: The movie was incredible, best I've seen this year.
Sentiment: positive
Text: Absolutely terrible, a waste of time.
Sentiment: negative
Text: {text}
Sentiment:

Multi-class — enumerate categories to cut common-token bias
Classify the topic of the following news article.
Categories: World, Sports, Business, Science/Tech
Article: {text}
Topic:

A pool should vary across dimensions: instruction framing, label verbalization, input position, and presence of few-shot examples. Homogeneous pools produce near-identical scores and give you nothing to choose between. For scenario tweaks: add explicit class definitions when the task is ambiguous; switch to CBM under class imbalance; use the instance-wise variant for per-instance label sets (multiple-choice QA); and use the domain's native terms as verbalizers ("benign"/"malignant", "compliant"/"non-compliant") when the model was trained on that vocabulary.

Implementation

The workflow from scratch: (1) define the task and verbalizers; (2) write 5–15 diverse templates covering direct, instructional, named-label, few-shot, question, and role-based styles; (3) collect unlabeled inputs (aim for n ≥ 50, ideally 100–200); (4) implement and run MI scoring; (5) optionally validate the winner against a tiny labeled set (20–50 examples) to sanity-check the proxy; (6) deploy the winner and only re-score when the distribution shifts.

The core mechanism is the MI score itself. Given a probability matrix of p(y | x, t), with optional CBM:

import numpy as np

def compute_mi_score(prob_matrix: np.ndarray, use_cbm: bool = True) -> float:
    """
    I(Y ; X | t) = H(Y | t) - H(Y | X, t)
    prob_matrix: shape [n_inputs, n_labels], rows are p(y | x, t).
    """
    if use_cbm:
        # Calibration by Marginalization: divide by marginal p(y|t), renormalize
        marginal = prob_matrix.mean(axis=0)
        prob_matrix = prob_matrix / (marginal[np.newaxis, :] + 1e-10)
        prob_matrix = prob_matrix / prob_matrix.sum(axis=1, keepdims=True)

    marginal_probs = prob_matrix.mean(axis=0)
    H_marginal = -np.sum(marginal_probs * np.log(marginal_probs + 1e-10))
    H_conditional = np.mean(
        -np.sum(prob_matrix * np.log(prob_matrix + 1e-10), axis=1)
    )
    return float(H_marginal - H_conditional)  # higher is better

Filling that matrix needs label probabilities per input. On the OpenAI Completions endpoint (the most practical proprietary path for MMI):

from openai import OpenAI
client = OpenAI()

def get_label_probs(prompt, labels, model="gpt-3.5-turbo-instruct"):
    """Top-5 logprobs at the next position -> softmax over the label set."""
    resp = client.completions.create(
        model=model, prompt=prompt, max_tokens=1, logprobs=5, temperature=0
    )
    top = resp.choices[0].logprobs.top_logprobs[0]
    lp = {}
    for label in labels:
        for surface in [label, label.lower(), " " + label, " " + label.lower()]:
            if surface in top:
                lp[label] = top[surface]
                break
        else:
            lp[label] = -100.0  # label fell outside top-5
    vals = np.array([lp[l] for l in labels])
    exp = np.exp(vals - vals.max())
    return exp / exp.sum()

Score every template the same way and take argmax. For open-source models (LLaMA, Mistral, Qwen, and the rest), Hugging Face Transformers gives full logit access and supports all-token scoring (sum log-probs over each label's tokens) — preferable for multi-token labels. Claude and Gemini expose no arbitrary logprobs; a forced-choice workaround (ask for exactly one label) yields one-hot distributions that degrade MI estimates badly, so for genuine MMI use a model with logprob access or a third-party provider that wraps one.

Configuration

Task-specific notes: for NLI use "entailment"/"neutral"/"contradiction" (or "yes"/"maybe"/"no"); for multiple-choice use "A"/"B"/"C"/"D" with the instance-wise variant; for more than 10 classes, the marginal-entropy ceiling is log(|Y|), so consider coarser groupings or instance-wise selection; for binary tasks, restrict logprobs to the two label tokens and renormalize.

Do: score on inputs from the actual deployment distribution; apply CBM; vary verbalization across templates, not just instruction wording; average MI over several random subsamples to cut variance; eyeball the winner's probability matrix to catch a degenerate template that scored well by accident.

Don't: fill the pool with near-synonymous phrasings (no discriminative signal); stack CC on top of MMI (use CBM); compare MI scores across different model families (rankings are model-relative); run the procedure per-request unless you genuinely need instance-wise selection; ignore logprob fallback handling (label tokens dropping out of top-5 corrupts the estimate — switch to all-token scoring).

Debugging

Testing and proving it works

The validation move is a holdout proxy check: after MMI picks a winner, evaluate it and the runner-up on a small labeled set (20–50 examples) to catch cases where MI misleads. To quantify MMI's value, the canonical metric is oracle-F1 recovery from Yang et al. (2024):

def oracle_recovery(mi_selected_f1, average_f1, best_f1):
    """Fraction of the average-to-best gap that selection recovers."""
    return (mi_selected_f1 - average_f1) / (best_f1 - average_f1) * 100
# Target: ~90%+ with MI_AGL + CBM

Two more metrics: regret (accuracy gap between the MMI pick and the oracle-best template; lower is better) and rank correlation (Spearman between MI scores and true accuracies on a labeled set; above 0.7 means strong alignment). When you're comparing MI variants (vanilla vs. CBM vs. MI_AGL) on the same labeled evaluation set, test whether the difference is real with McNemar's test on each variant's correct/incorrect predictions. For an honest A/B, split a labeled set into a 30% selection set and 70% evaluation set, run MMI on the selection set (ignoring its labels) versus validation-set selection (using them), and compare evaluation accuracy — the gap is the cost of going unsupervised. Stop expanding the pool when the top-two MI gap is clear (above ~0.05 nats) or three new templates fail to beat the best.

Limitations and constraints

1. Closed-vocabulary only. You must enumerate every label to compute p(y | x, t). Free-form output spaces are intractable — this is structural, not an engineering gap.
2. The MI-accuracy link is empirical, not guaranteed. The 90% gap-closure figure is an average across 8 datasets. It can break on tasks with spurious surface-label correlations, near-tied templates that diverge in accuracy, or tasks at the edge of the model's competence.
3. Logprob dependency. Available from OpenAI Completions, Hugging Face, Together AI, Fireworks, Anyscale, Replicate; not from Anthropic Claude chat, Google Gemini chat, or most proprietary chat-first APIs.
4. Verbalizer sensitivity. "positive"/"negative" vs. "good"/"bad" vs. "pos"/"neg" give different scores, and MI alone can't tell you the optimal verbalizer — it stays an external design choice.
5. Selection, not generation. Given only weak templates, MMI returns the least-bad one. It can't invent a better prompt; pair it with an LLM generator for that.
6. Capability floor. If every template yields near-uniform distributions, both entropies max out, MI ≈ 0, and the ranking is meaningless.

It degrades gracefully: scores get noisier but stay directionally useful above n ≈ 20, and shouldn't be trusted below n = 10. Watch the edge cases — ambiguous inputs raise conditional entropy and legitimately lower MI (the model genuinely hasn't resolved them); a template whose few-shot demos contradict its instruction can score high on MI yet be wrong on accuracy; out-of-domain scoring inputs can scramble the ranking; and extreme class imbalance suppresses marginal entropy unless you apply CBM. A badly miscalibrated, overconfident model collapses the conditional-entropy term, leaving MI dominated by class balance alone.

Advanced techniques

Instance-wise selection (MI_AGL). Rather than one global template, compute MI per test input and pick the best template for that input. This is the variant that reaches 94.98% (and 96.85% with CBM), at the cost of running multiple templates per request.

MMI for chain-of-thought. A 2024 ACL Findings paper ("Learning to Maximize Mutual Information for Chain-of-Thought Reasoning") applies MI between the reasoning trace and the input, not just the final label. Collect the chain as the output, discretize it (hash the first tokens, or cluster semantically), and pick the CoT template that produces the most input-specific reasoning. Most useful where reasoning genuinely varies, not for shallow binary tasks.

Structured output. MMI can rank templates that enforce different formats (JSON keys, fixed fields); compute MI over the finite set of extracted field values. Tighter formats tend to score higher because they lower conditional entropy.

Hybrids that work well: MMI + APE — let APE generate 20–50 candidates, then score them all with MMI instead of labeled validation; MMI + self-consistency — MMI picks the template, self-consistency (temperature > 0, majority vote) cuts inference variance; MMI + DSPy — plug MI scoring in as a custom metric inside MIPROv2 for unsupervised optimization; MMI + RAG — apply MMI twice, once for the retrieval-query template and once for the answer-synthesis template, both label-free.

Risk and ethics

Two failure modes deserve naming. Silent miscalibration: the model is confident and diverse but consistently wrong — MMI happily selects the template that fails most decisively (high confidence on wrong labels reads as low conditional entropy). This is most common with small models on out-of-domain tasks. Verbalizer gaming: a template framed in vocabulary that coincides with common next-token completions scores artificially high even when the classification logic is off. On the positive side, the procedure is fully transparent and auditable — every score is reproducible — though in regulated domains (medical, legal, financial) an automatically selected prompt may still need documentation for compliance. MMI selects before deployment and offers no defense against runtime prompt injection through the {text} slot; delimit user input and add instruction-following resistance regardless of MI score.

Ecosystem

The reference implementation is unified-prompt-selection (github.com/soheeyang/unified-prompt-selection, Yang et al., 2024). It covers all MI variants (MI_G, MI_L, MI_GL, MI_A, MI_AGL), baselines (GE, MDL, ZLP, ZPM, ZMV, PPL), both CBM and CC calibration, and runs across the paper's 10 decoder models (1.3B–66B) and 13 datasets. DSPy (dspy.ai) ships MIPROv2 (arXiv:2406.11695), a Bayesian-optimization optimizer reporting up to 13% gains over hand-crafted prompts; it uses task-metric evaluation but accepts MI as a custom metric. LangChain + LangSmith supply template management and evaluation hooks you can wire an MI scorer into (no built-in scorer). The OpenAI Completions endpoint (gpt-3.5-turbo-instruct, logprobs=5) is the practical proprietary path; Together AI, Fireworks, Anyscale expose logprobs for open models via API; Hugging Face Transformers + TGI is the canonical open-source route with native all-token scoring. For complementary evaluation infrastructure, PromptBench (github.com/microsoft/promptbench, arXiv:2312.07910, JMLR 2024), AutoPrompt (github.com/Eladlev/AutoPrompt), and HELM / EleutherAI lm-evaluation-harness can be repurposed to score by MI rather than accuracy.

To migrate from manual selection: inventory your current templates, expand to a 5–15 pool by varying phrasing, verbalization, and few-shot content, collect 50–100 unlabeled inputs from production logs, run MMI, and A/B-test before switching if it picks something new. Re-score whenever the model version changes (gpt-3.5-turbo-0125 → a newer snapshot can shift rankings) or the input distribution drifts — track the marginal label distribution and re-run when it moves (KL above ~0.1 nats signals meaningful drift). Log template content, MI scores, sample metadata, and model version for rollback.

Future directions

The frontier is pushing MI past single-label classification. Generation evaluation via output clustering (discretize the output space, then compute MI between clusters and inputs) could extend it to summarization and QA. Multi-modal selection applies the same recipe to image-text prompt templates. MI-guided template synthesis would close the loop — generate candidates, score by MI, and use the differential to steer generation toward higher-MI prompts. Personalized instance-wise selection could choose templates per user, not just per input. And MI is already moving from a selection signal to a training objective: InfoPO (NAACL 2025, arXiv:2505.08507) replaces the Bradley-Terry preference model with a direct MI objective for alignment. Open questions remain — when and why the MI-accuracy correlation breaks down at the instance level, whether the verbalizer can be jointly optimized unsupervised, how to scale MI to 10–100-class output spaces affordably, whether a theoretical bound on the oracle gap exists, and how native reasoning models (o1, o3, Gemini 2.5 Pro) change the probability distributions MI reads. See also the optimization-perspective survey of automatic prompt engineering (arXiv:2502.11560).

References

Primary papers:

• Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. "An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels." ACL 2022, Dublin, Ireland, pages 819–862. arXiv:2203.11364
• Yang, S., Kim, J., Jang, J., Ye, S., Lee, H., and Seo, M. "Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis." TACL 2024 / ACL 2024. arXiv:2305.14877
• Zhao, T., Wallace, E., Feng, S., Klein, D., and Singh, S. "Calibrate Before Use: Improving Few-Shot Performance of Language Models." ICML 2021. arXiv:2102.09690

Foundations and context:

• Holtzman, A., West, P., Shwartz, V., Choi, Y., and Zettlemoyer, L. "Surface Form Competition: Why the Highest Probability Answer Isn't Always Right." EMNLP 2021. arXiv:2104.08315
• Perez, E., Kiela, D., and Cho, K. "True Few-Shot Learning with Language Models." NeurIPS 2021. arXiv:2105.11447

Related optimization methods:

• Zhou, Y., et al. "Large Language Models Are Human-Level Prompt Engineers." ICLR 2023. arXiv:2211.01910 (APE)
• Yang, C., et al. "Large Language Models as Optimizers." arXiv:2309.03409 (OPRO)
• Pryzant, R., et al. "Automatic Prompt Optimization with 'Gradient Descent' and Beam Search." EMNLP 2023. arXiv:2305.03495 (ProTeGi)
• Prasad, A., et al. "GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models." EACL 2023. arXiv:2203.07281
• Deng, M., et al. "RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning." EMNLP 2022. arXiv:2205.12548

Extensions and tooling (2024–2025):

• Xiao, T., et al. "InfoPO: On Mutual Information Maximization for Large Language Model Alignment." NAACL 2025. arXiv:2505.08507
• "Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation." arXiv:2411.07773
• "Automatic Prompt Selection for Large Language Models." arXiv:2404.02717 (APS)
• Khattab, O., et al. DSPy MIPROv2. arXiv:2406.11695
• Zhu, K., et al. "PromptBench: A Unified Library for Evaluation of Large Language Models." JMLR 2024. arXiv:2312.07910
• "A Survey of Automatic Prompt Engineering: An Optimization Perspective." arXiv:2502.11560

Summary

• What it is: an unsupervised, information-theoretic procedure that ranks prompt templates by the mutual information between inputs and the model's label predictions — no ground-truth labels required.
• Why it works: a good template makes outputs both diverse across inputs (high marginal entropy) and confident per input (low conditional entropy), which is exactly what task-solving behavior looks like.
• When to use it: closed-vocabulary classification, multiple candidate templates, no labels, and an API that exposes token-level logprobs.
• When not to: free-form generation, a single template, sub-10 input samples, continuous labels, or logprob-less chat APIs.
• What it costs: a one-time k×n×|Y| calls to score (often a dollar or two); zero per-request overhead for a single global template.
• What to remember: use CBM (not CC) for calibration, all-token scoring for multi-token labels, larger well-calibrated models for the tightest MI-accuracy link, and a fairness audit on the winner — MI rewards confident diversity, not fairness.
• The numbers: ~90% oracle-gap closure unsupervised (Sorensen 2022); 87.79% → 94.98% → 96.85% oracle F1 as you add all-token, instance-wise, and CBM (Yang 2024); MDL by contrast recovers only 20–40%.