Universal self-adaptive prompting (USP): a complete guide

Few-shot prompting works because examples guide the model, but writing and labeling those examples by hand is slow and you often have none. USP flips that: it lets the model write its own demonstrations from unlabeled data, picking the answers it was most confident about and reusing them as the few-shot examples. Across more than 40 tasks on PaLM and PaLM 2, USP beats standard zero-shot by wide margins and often matches or beats human-labeled few-shot (Wan et al., EMNLP 2023).

See it work

Say you're classifying support tickets as billing, bug, or feature-request, and you have a pile of unlabeled tickets but no gold labels.

Plain zero-shot is brittle on the borderline ones:

Prompt:
Classify this ticket: "App keeps logging me out after the latest update."
Categories: billing, bug, feature-request

Output: feature-request   ← wrong, and the model wasn't sure

USP runs the model over your unlabeled pile first, keeps the tickets it answered most confidently, and pastes those back as examples:

Prompt (USP-assembled, pseudo-demos chosen by confidence):
Ticket: "Charged twice for one month."        → billing
Ticket: "Export button does nothing on click." → bug
Ticket: "Add dark mode please."                → feature-request

Classify this ticket: "App keeps logging me out after the latest update."
Output: bug   ← correct; the in-distribution examples anchored the label space

Same model, same unlabeled data, no human labels. The only new ingredient is the model's own confident guesses, recycled as guidance.

The mental model

Think of a new hire who's never seen your ticket queue. Hand them nothing and they guess by vibes. Instead, let them triage a stack on their own, pull the five tickets they were most sure about, and pin those to the wall as a cheat sheet. Now every later call is anchored to clear, self-made reference points.

USP is in-context learning with no labels: the model grades its own zero-shot answers, keeps the confident ones, and uses them as the demonstrations it never had.

How it works

USP needs only a small pool of unlabeled queries and an inference-only LLM (no gradients, no labels). It runs in two stages: build a demo set once, then prompt with it.

1. Gather unlabeled queries. A handful of in-distribution inputs is enough; no answers required.
2. Pick the task type. USP sorts every NLP task into one of three buckets: classification (CLS), short-form generation (SFG), and long-form generation (LFG). The bucket decides which scorer runs.
3. Generate candidate answers. Run the model zero-shot over the pool. For CLS one pass yields a label distribution; for SFG and LFG, sample several responses per query.
4. Score confidence with the matching selector. Each task type uses a different confidence proxy (see below). Low uncertainty is the signal.
5. Select the top demonstrations. Rank by confidence, take the best (the paper uses 5 pseudo-demos), favoring diversity so the set isn't redundant.
6. Prompt with the frozen demos. Prepend the chosen query→answer pairs to every test input and run one normal pass. No test-time voting needed.

Why it works

The core bet: a model's confident answers are more likely to be correct, and correct in-distribution examples make good few-shot demonstrations. The paper validates this "confident predictions are more likely better" assumption across all three task types.

Where it shines

USP is built to be universal, so it spans the three task families it defines:

• Classification (CLS): sentiment, topic, intent, NLI. Reported around +8.2% accuracy over zero-shot on PaLM-540B, and 73.80% vs 73.28% for a 5-shot human-labeled baseline.
• Short-form generation (SFG): extractive QA, arithmetic and commonsense reasoning, closed-book answers. Reported roughly +15.8% exact-match over zero-shot.
• Long-form generation (LFG): summarization, translation, free-form generation. Reported about +29% ROUGE-1 over zero-shot.

The headline: across 40+ NLU, NLG, and reasoning tasks, USP is "considerably stronger" than zero-shot and "often comparable to or even superior to" few-shot that uses human labels. On BIG-Bench Hard it matched 3-shot handcrafted demonstrations on many tasks.

When to use it (and when not)

Reach for USP when: you have unlabeled in-domain inputs but no labels, you can only call the model (black-box, inference-only), and zero-shot underperforms. It's ideal when labeling is expensive or impossible.

Skip it when: you already have solid labeled demonstrations (just do few-shot), the task is trivial enough that zero-shot is fine, or you have no representative unlabeled inputs to mine.

Model fit: USP was validated on large instruction-capable models (PaLM, PaLM 2). The confidence-equals-correctness assumption leans on a model already decent at the task zero-shot; very small or weak models give noisy confidence signals and weaker demos.

Escalation: if confident answers still aren't accurate enough, or you accumulate real labels, graduate to human-labeled few-shot or fine-tuning.

Variants and alternatives

Structure and components

A USP setup has three moving parts:

• A small unlabeled query pool drawn from the target task's input distribution.
• A task-type tag (CLS, SFG, or LFG) that routes to the right scorer.
• A pseudo-demo selector with a confidence scoring function per task type.

The assembled prompt is ordinary in-context learning. Only the demo sourcing is special:

[pseudo-demo 1: query → model's confident answer]
[pseudo-demo 2: query → model's confident answer]
...
[pseudo-demo k]

[test query]

The scoring functions

Each task type defines uncertainty over its own output shape. Confident (low-uncertainty) candidates win.

import numpy as np
from collections import Counter

def cls_score(class_probs):
    # Classification: negative entropy of the label distribution.
    # Peaked distribution = confident = high score.
    p = np.asarray(class_probs)
    return -(-(p * np.log(p + 1e-12)).sum())

def sfg_score(sampled_answers):
    # Short-form generation: self-consistency across sampled answers.
    # More agreement on one answer = lower entropy = higher score.
    counts = np.array(list(Counter(sampled_answers).values()), dtype=float)
    p = counts / counts.sum()
    return -(-(p * np.log(p + 1e-12)).sum())

def lfg_score(sampled_texts, rouge_fn):
    # Long-form generation: exact matches are unlikely, so use the
    # average pairwise ROUGE overlap among samples as a consistency proxy.
    pairs = [(i, j) for i in range(len(sampled_texts))
             for j in range(i + 1, len(sampled_texts))]
    return np.mean([rouge_fn(sampled_texts[i], sampled_texts[j]) for i, j in pairs])

The selection algorithm

def build_pseudo_demos(pool, task_type, llm, k=5):
    scored = []
    for query in pool:
        if task_type == "CLS":
            probs = llm.classify(query)          # one pass, label distribution
            answer = probs.argmax()
            score = cls_score(probs)
        elif task_type == "SFG":
            samples = llm.sample(query, n=5)     # several stochastic decodes
            answer = Counter(samples).most_common(1)[0][0]
            score = sfg_score(samples)
        else:  # LFG
            samples = llm.sample(query, n=5)
            answer = samples[0]
            score = lfg_score(samples, rouge_1)
        scored.append((score, query, answer))

    # Rank by confidence, then take a diverse top-k as demonstrations.
    scored.sort(reverse=True, key=lambda x: x[0])
    return select_diverse(scored, k)

Configuration

Implementation workflow

1. Collect a small set of unlabeled, representative inputs.
2. Decide the task type (or detect it from the expected output shape).
3. Run zero-shot inference over the pool, sampling multiple times for SFG/LFG.
4. Score every candidate with the matching confidence function.
5. Select the top-k confident, diverse query→answer pairs.
6. Freeze them as the demonstration block.
7. Prepend the block to each test query and run a single pass.

Do: keep the pool in-distribution, sample enough for stable consistency estimates, and favor diversity so demos cover the space.

Don't: mix task types in one selector, trust confidence from a weak base model, or skip the diversity step and end up with five near-identical demos.

Debugging

Testing and how to prove it

Compare three conditions on a labeled holdout: zero-shot, USP, and (if you have labels) human few-shot. Use the metric that fits the task type.

def evaluate(test_set, llm, demos=None, metric=accuracy):
    preds = [llm.predict(x.input, demos=demos) for x in test_set]
    return metric([x.gold for x in test_set], preds)

zs   = evaluate(test_set, llm, demos=None)              # zero-shot baseline
usp  = evaluate(test_set, llm, demos=usp_demos)         # USP pseudo-demos
fs   = evaluate(test_set, llm, demos=human_demos)       # few-shot upper bound

print(f"zero-shot {zs:.3f} | USP {usp:.3f} | few-shot {fs:.3f}")

Metrics by type: accuracy/F1 for CLS, exact-match for SFG, ROUGE for LFG. The result you're hoping to see is USP closing most of the gap between zero-shot and few-shot, without a single human label.

Limitations

• Confidence isn't correctness. When the model is confidently wrong, USP happily promotes bad demos. The whole method rests on that correlation holding.
• Weak models break it. Small or poorly-aligned models give unreliable confidence, so demos are noisy.
• Needs representative inputs. Off-distribution or tiny pools yield poor candidates.
• Setup overhead. The one-time pass over the pool (plus extra samples for SFG/LFG) costs compute before any test query runs.
• Task-type assignment. A task that doesn't fit cleanly into CLS, SFG, or LFG needs a judgment call on which scorer to use.

How it relates to COSP

USP is the generalization of COSP (Consistency-based Self-adaptive Prompting, Wan et al., ACL 2023 Findings). COSP targeted chain-of-thought reasoning only, scored demos by consistency, entropy, repetition, and diversity, and relied on majority voting at test time, lifting zero-shot reasoning by up to 15% on PaLM-62B, PaLM-540B, and GPT-3 across six reasoning datasets (MultiArith, GSM8K, AddSub, SingleEq, CommonsenseQA, StrategyQA). USP keeps the confidence-selection idea but extends it to all three task types and drops test-time voting, so it covers classification and generation too and stays a single pass at inference.

Summary

• USP automates few-shot prompting with zero labels: the model's own confident zero-shot answers become the demonstrations.
• It sorts every task into classification, short-form generation, or long-form generation, then scores confidence with a matching function (negative entropy, self-consistency, pairwise ROUGE).
• It needs only a small unlabeled pool and an inference-only LLM; pick the top ~5 confident, diverse candidates as pseudo-demos.
• Across 40+ tasks on PaLM and PaLM 2, USP beats zero-shot by wide margins and often matches or beats human-labeled few-shot.
• It generalizes COSP from reasoning-only with test-time voting to all task types with a single inference pass.
• Reach for it when you have unlabeled in-domain data and no labels; skip it when you already have good labeled demos or the base model is too weak to trust its own confidence.