Demonstration ensembling (DENSE): a complete guide

Few-shot prompting has a dirty secret: which examples you pick decides everything. Swap one demonstration set for another and the same model on the same task can swing from near-random to state-of-the-art. Demonstration Ensembling (DENSE) stops fighting that instability and exploits it instead. You run the query through several diverse demonstration sets, then aggregate the answers, the way a panel of judges beats any single judge. On sentiment analysis that buys roughly 15-25% accuracy over a single set, and up to 30% on compositional reasoning.

The flip is simple. Instead of hunting for the one perfect set of examples that doesn't exist, you embrace many imperfect sets and let voting cancel their individual mistakes. Diversity becomes the feature, not the bug.

See it work

Take an ambiguous review and a single demonstration set focused on explicit sentiment words. The model latches onto "love" and guesses wrong.

Review: "Finally, a product that doesn't break immediately!"

Single set (explicit-words examples)  → negative   (it saw "break")

DENSE, 5 diverse sets:
  Set 1 (explicit words)     → negative
  Set 2 (sarcasm/nuance)     → positive
  Set 3 (context-dependent)  → positive
  Set 4 (mixed sentiment)    → positive
  Set 5 (domain examples)    → positive

Majority vote → positive   (4 of 5)   confidence 0.8

One set anchored on the word "break" and missed the relief in the sentence. Four sets that had seen context-dependent phrasing caught it. The vote rescues the answer, and the 4-of-5 split doubles as a confidence signal.

The mental model

Think of a hiring panel. One interviewer has blind spots; five interviewers with different backgrounds catch each other's mistakes, and when they all agree you trust the call more. DENSE is that panel, where each "interviewer" is a different set of examples priming the model toward a different slice of the task.

DENSE turns the variance of demonstration selection from a liability into a voting population.

This is ensemble learning applied to in-context learning. Classical ensemble theory says combining weak learners yields a strong one when the learners are accurate (better than chance) and diverse (they make different errors). Each demonstration set is a weak learner; aggregation does the rest.

How it works

DENSE is single-pass and multi-stage: you prepare once, then for each query you fan out to K sets, infer, and aggregate. No feedback loop back into selection (the iterative variants add one).

1. Build a demonstration pool. Collect 20-100 correct, well-formatted examples with natural diversity in length, style, edge cases, and domain. Quality is non-negotiable: one bad example pollutes any set it lands in.
2. Pick a diversity strategy. Clustering-based (group similar demos, sample one per cluster), feature-based (maximize spread over length/complexity/style), coverage-based (cover the input space), or plain random sampling as a baseline.
3. Generate K sets of 3-10 demonstrations each, keeping the instruction identical across sets so only the examples vary.
4. Construct K prompts (instruction + set + test instance) and run K inferences, in parallel when you can.
5. Parse and aggregate. Majority vote for the common case; weighted or confidence-based voting when sets differ in quality; verification scoring for high stakes; mean for numeric outputs.
6. Estimate confidence from agreement: proportion agreeing, entropy, or the margin between the top two vote counts. Low agreement flags an ambiguous instance.

Why it works

Ablation studies and empirical work rank the levers like this:

The causal chain underneath: diverse sets produce uncorrelated errors, so voting cancels the random ones (error decorrelation). If individual sets hit 70% accuracy with uncorrelated errors, an ensemble of five can reach 85-90%. Diversity also widens coverage of the task space, cancels idiosyncratic biases, and helps the model abstract the core concept rather than memorize one set's surface patterns. Park et al. (2025) call this last effect "beyond coverage": diversity helps even when the test instance was already covered.

Where it shines

DENSE pays off wherever demonstration choice swings results, ambiguity is real, or multiple valid perspectives exist.

• Classification. Sentiment, topic, intent, content moderation: 15-25% accuracy, strongest on boundary cases. One sentiment setup went from a 72% single-set baseline to 87-92% with five diverse sets plus majority voting (at 5x the API calls); one topic study cut misclassification from 28% to 11%.
• Reasoning. Arithmetic on GSM8K rose from 65% (single set, GPT-3.5) to 77-78%; CommonsenseQA gained 10-15%; benefit grows with problem complexity.
• Generation. Summarization improved ROUGE-L by 23% and cut hallucination 30% via cross-set verification; creative writing gained 35% in human preference.
• Structured output and extraction. Data extraction errors dropped 28%; format violations dropped 25-35%.

Domain results, all over single-set baselines:

• Medical. Clinical NER +8-12% F1, relation extraction +15-20%, clinical-note summarization -25% factual errors. Medical QA +12-18% with a 40% cut in critically wrong answers (Wu et al., 2025, LLM-Synergy: boosting-based weighted majority vote with cluster-based dynamic model selection, across three medical QA datasets).
• Legal. Contract clause classification +20%, risk assessment +18%, case-law relevance +15% nDCG@10.
• Code. Python Pass@1 went 45% to 61% (+16 points), syntax errors -35%, functional correctness +22%; GitHub Copilot research (2024) saw a 20% syntax-error reduction and 15% functional-correctness gain from diverse code examples. Code translation +18%, style consistency +40%.
• SQL. SSEV (single-agent self-refinement with ensemble voting) using weighted majority voting hit 85.5% execution accuracy on Spider 1.0-Dev (vs 76% baseline), 86.4% on Spider 1.0-Test, and 66.3% on BIRD-Dev (up from a 57% baseline), with adaptive weighting beating simple majority by 7-9 points (Liu et al., January 2025).
• Scientific. Abstract screening +15% recall while holding precision, with 60% fewer false positives needing human review; methodology identification +17% F1.
• Financial. Stock-related sentiment +28% (domain language benefits more than general), earnings-call analysis +23%, credit-risk +14%, fraud detection +19% precision at constant recall.

Versus alternatives: 40-60% over zero-shot; 15-25% over single-set few-shot with 30-40% less variance; comparable to fine-tuning but on 10-100 examples instead of 1000+; and 8-15% additional gain when stacked on RAG. By size, small models (1-7B) gain 10-15%, medium (7-30B) 15-25% at the best cost ratio, large (30-100B+) 20-35%. Improvements were statistically significant (p below 0.01) in 87% of benchmarks tested, with Cohen's d of 0.6-1.2, across GPT, Claude, Llama, and PaLM families.

When to use it (and when not)

Reach for DENSE when a task is ambiguous (human agreement below 80%), demonstration choice swings accuracy more than 15 points, errors cost more than 10x an inference, or you need consistency on high-stakes calls. It also shines when you want output diversity (brainstorming, creative work) and when you're new to a domain and unsure which examples are best.

Skip it when the task is deterministic or regex-solvable, a single set already clears 95%, latency must stay under 500ms even with parallelism, you have 10,000+ labeled examples (fine-tune), the model is below 7B parameters, or pilot testing shows under 5% gain.

Model fit. Minimum is 7B parameters with a 4K-token context and reliable instruction following. Recommended is 30B+ with 8K context for the best cost-benefit. Optimal is 70B+ with 32K context plus reasoning and structured-output support; GPT-4, Claude 3 Opus/Sonnet, Gemini Pro, and Llama 3 70B+ all qualify. Models below 7B extract too little from diversity to justify the cost.

Escalate to fine-tuning past 10,000 stable examples or 100K+ queries; to RAG when knowledge (not demonstration choice) is the bottleneck or the pool exceeds 1000 examples; to model ensembling when DENSE lands at 80-90% but you need 95%+.

How DENSE differs from its neighbors: self-consistency varies reasoning paths with the same demonstrations (pick it when reasoning variance dominates); prompt ensembling varies the whole instruction (pick it when phrasing dominates); retrieval-based ICL picks the most similar demos per query (pick it for pools above 100 where similarity beats diversity). DENSE keeps the instruction fixed and varies the examples, deliberately favoring diversity over similarity for moderate pools of 20-100.

Structure and components

Five pieces are required, the rest are optional polish.

Required: the demonstration pool (15-20 examples minimum, 50-100 optimal, all 100% correct); a diversity strategy; an aggregation method; a consistent instruction template; and a prompt constructor that assembles instruction + set + test instance.

Optional but recommended: a confidence estimator (agreement rate, entropy, or margin), and logging of every output, agreement rate, and confidence for later analysis. Optional: adaptive ensemble sizing, a verifier model for ties and filtering, and a caching layer for repeated queries.

The prompt format stays parallel and explicitly labeled across every set:

{instruction}                      # identical for all K sets

Input: {demo 1 input}
Output: {demo 1 output}

Input: {demo 2 input}
Output: {demo 2 output}
...

Input: {test_input}
Output:

Design principles that matter most: keep the instruction and format consistent across sets (don't mix JSON and plain text unless format diversity is the point); keep diversity within bounds of task relevance (meaningful semantic spread, never off-topic noise or incorrect patterns); make output format explicit; and give each set at least one clear prototypical example, not only edge cases.

The core algorithm

The whole technique fits in one sketch: sample diverse sets, infer, aggregate.

def dense(test_input, pool, instruction, k=5, n_demos=5,
          strategy="cluster", aggregation="majority_vote", temperature=0.3):
    # 1. K diverse demonstration sets from the pool
    demo_sets = sample_diverse_demonstrations(pool, k, n_demos, strategy)

    # 2. one inference per set, instruction held constant
    outputs = []
    for demo_set in demo_sets:
        prompt = construct_prompt(instruction, demo_set, test_input)
        outputs.append(parse(llm(prompt, temperature=temperature)))

    # 3. aggregate to a single answer plus a confidence signal
    return aggregate(outputs, method=aggregation)

Aggregation is where the ensemble earns its keep. Majority voting is the baseline; weighted voting (by demonstration quality or model confidence) and confidence-based selection are the upgrades worth 7-9 points:

from collections import Counter

def aggregate(outputs, method="majority_vote", weights=None):
    if method == "majority_vote":
        counts = Counter(outputs)
        answer = counts.most_common(1)[0][0]
        return answer, counts[answer] / len(outputs)        # answer, confidence

    if method == "weighted_vote":
        weights = weights or [1.0] * len(outputs)
        tally = {}
        for out, w in zip(outputs, weights):
            tally[out] = tally.get(out, 0) + w
        answer = max(tally, key=tally.get)
        return answer, tally[answer] / sum(weights)

A concrete platform wiring (Claude), kept to one class rather than one per provider:

import anthropic, random
from collections import Counter

class DenseClaude:
    def __init__(self, model="claude-sonnet-4-6", k=5, n_demos=5):
        self.client = anthropic.Anthropic()
        self.model, self.k, self.n_demos = model, k, n_demos

    def execute(self, test_input, pool, instruction, temperature=0.3):
        demo_sets = [random.sample(pool, self.n_demos) for _ in range(self.k)]
        outputs = []
        for demo_set in demo_sets:
            prompt = instruction + "\n\n"
            for d in demo_set:
                prompt += f"Input: {d['input']}\nOutput: {d['output']}\n\n"
            prompt += f"Input: {test_input}\nOutput:"
            msg = self.client.messages.create(
                model=self.model, max_tokens=100,
                temperature=temperature,
                messages=[{"role": "user", "content": prompt}])
            outputs.append(msg.content[0].text.strip())
        counts = Counter(outputs)
        answer = counts.most_common(1)[0][0]
        return answer, counts[answer] / len(outputs), outputs

Configuration

Token budgeting: a single set runs 570-3700 tokens; a full K=5 query lands around 3000-23500 tokens, so budget 5000-10000 per query for typical tasks. Latency is K x 2-5s sequential (10-25s at K=5) but collapses to 2-5s in parallel; a K=3 parallel setup with caching hits 1-2 seconds for near-real-time work.

Implementation workflow

Plan for 10-20 hours end to end, and start simple.

1. Define success (metrics, acceptable latency and cost, critical error types) and build a baseline single-set few-shot to beat. Aim for a 15-25% gain.
2. Gather 20-100 examples, validate every one for correctness, tag metadata (length, complexity, category) so feature-based diversity is possible.
3. Build the pipeline at K=5, n_demos=5, majority voting, with logging from day one. Test on 100-500 validation examples.
4. Tune K (3/5/7/10), n_demos, temperature, diversity strategy, and aggregation; refine the pool toward poorly covered cases. Split data 60/20/20 train/val/test and touch test only once.
5. Productionize with parallel calls and caching, then A/B against the baseline and roll out gradually (10% to 50% to 100%) with a rollback ready.

Do: start simple and add complexity only when it pays; demand 100% demonstration correctness; log all K outputs and agreement rates; validate on held-out data with significance testing. Don't: over-complicate aggregation before majority voting fails; ship low-quality demos (one bad example hurts the whole ensemble); over-fit the validation set; ignore the cost multiplier; deploy without A/B testing.

Debugging

Testing

Validate on a true holdout (never let validation or test instances appear as demonstrations), or k-fold cross-validation when data is scarce. Cover happy path (40%), edge cases (30%), boundary conditions (20%), and adversarial inputs (10%, including paraphrases, negations, and sarcasm). Track task-specific metrics (accuracy/F1, ROUGE and BERTScore, exact match, Pass@k, schema validity) plus a DENSE-specific consistency score: agreement above 0.8 is confident, 0.6-0.8 moderate, below 0.6 signals an ambiguous task or a problematic pool.

Prove it beats the baseline with a paired comparison rather than eyeballing:

import numpy as np
from scipy import stats

def compare(baseline_scores, dense_scores, alpha=0.05):
    t_stat, p = stats.ttest_rel(dense_scores, baseline_scores)   # paired
    diff = np.mean(dense_scores) - np.mean(baseline_scores)
    cohens_d = diff / np.std(dense_scores + baseline_scores)
    return {"improvement": diff, "p_value": p,
            "significant": p < alpha, "effect_size": cohens_d}

Read p below 0.05 as a real difference and Cohen's d above 0.5 as a medium-or-larger effect; insist the mean improvement clears the cost multiplier before shipping.

Limitations

Some constraints are structural, not bugs to fix:

• Cost multiplier. K inferences per query, full stop. Adaptive sizing and caching reduce the average, never the floor.
• Diminishing returns. Gains plateau after 5-7 sets; K=10 to K=15 typically adds under 2%. Don't exceed K=10 without a reason.
• Garbage in, garbage out. DENSE diversifies and aggregates; it adds no information absent from the demos. A bad pool stays bad.
• No free capability. It can't exceed the base model's ceiling. If the model can't do the task with perfect demos, ensembling won't save it.
• Task unsuitability. If a single set already clears 95%, or the task is deterministic, diversity is just noise.
• Latency floor. Even parallel, you wait for the slowest of K generations: roughly 1-2 seconds versus 0.5-1 for a single call, which rules out sub-500ms use.

Advanced and hybrid techniques

Stack DENSE with the techniques it composes well with. DENSE + chain-of-thought puts reasoning steps in each demonstration for complex reasoning. DENSE + self-consistency varies demonstrations and sampling for maximum reliability. DENSE + RAG handles knowledge-intensive and demonstration-sensitive tasks at once. DENSE + verification adds a scorer for high-stakes validation. For ambiguity, widen K to 7-10 and surface disagreement as a signal; for data-scarce settings, reuse demonstrations across sets in different combinations to manufacture diversity from a small pool. An unconventional payoff: feeding agreement-filtered DENSE outputs back as fine-tuning data yields 20-30% better models than single-source data.

Risk and ethics

DENSE generally reduces idiosyncratic bias because diverse sets cancel each other's framing, and ensemble methods calibrate confidence better than single models. But it can't fix a pool that's biased in one direction; correlated bad demos produce correlated bad votes. In high-stakes domains (medical, legal, financial), pair it with confidence thresholds, expert-validated demonstrations, human review for low-agreement cases, and full output logging for audit trails. Validate input to blunt prompt injection, since K prompts mean K attack surfaces.

Ecosystem

DENSE rides on standard few-shot tooling: LangChain's FewShotPromptTemplate, plus OpenAI and Anthropic SDKs, wrapped in a sampling-plus-aggregation loop. It sits in the ensemble-prompting family next to self-consistency, prompt ensembling, and DiVeRSe/AMA-style verifier voting. Transition into it from single-set few-shot by adding a sampling loop and a vote; transition out toward fine-tuning or RAG when data volume or knowledge needs outgrow it. Open questions remain around optimal diversity metrics, smarter aggregation past majority voting, and dynamic per-instance ensemble sizing.

Summary

• DENSE runs a query through K diverse demonstration sets and aggregates the answers, turning demonstration-selection instability into a voting population.
• Headline gains: 15-25% over single-set few-shot, up to 30% on compositional reasoning, 40-60% over zero-shot, significant (p below 0.01) in 87% of benchmarks tested.
• Effectiveness is driven most by demonstration relevance (35-40%) and diversity quality (25-30%); aggregation method adds 7-9 points; gains plateau past 7-10 sets.
• Default to Standard DENSE (K=5, weighted voting, ~5x cost, 18-25% gain); scale to K=7-10 for high-stakes, drop to K=3 or adaptive when cost or latency bites.
• Skip it for deterministic tasks, single sets already above 95%, sub-500ms latency, 10,000+ labeled examples, or models below 7B parameters.
• The cost is structural: K inferences per query, a 3-10x multiplier you can soften with caching and adaptive sizing but never erase.