Prompt mining: a complete guide

Your model knows the answer — it just can't always hear the question. Ask "Barack Obama is a _ by profession" and it fumbles; ask "Barack Obama worked as a _" and it answers cleanly. Same fact, same model, different wording — and the wording, not the knowledge, was the bottleneck the whole time. Prompt mining stops guessing at phrasings: it harvests the way a giant corpus actually states a relationship, keeps the templates the model responds to best, and averages across them. That one move lifted LAMA knowledge-probing accuracy from 31.1% to 39.6% on BERT-base (Jiang et al., 2020).

See it work

You want the profession a model has stored for someone. Watch the same [MASK] get filled two ways.

Manual:  "Barack Obama is a [MASK] by profession."
  top fills: senator (0.18), lawyer (0.14), politician (0.11)  ← spread thin, unsure

Mined:   "Barack Obama worked as a [MASK]."   (lifted from Wikipedia phrasing)
  top fills: lawyer (0.41), professor (0.12), senator (0.09)  ← mass on the right answer

Nothing about the model changed. The manual prompt smears probability across plausible-sounding wrong answers; the mined prompt — phrased the way the corpus actually describes professions — concentrates it on the truth. That's the whole game: find the question the model is primed to answer. Across LAMA, swapping hand-written templates for mined ones moved micro-averaged accuracy up several points, and ensembling many mined prompts moved it further.

The mental model

Think of a model's knowledge as a person who only answers if you phrase the question the way they're used to hearing it. They know Obama was a lawyer — but they freeze on "is a _ by profession" and answer instantly to "worked as a _." You don't teach them anything new; you just find the question they're primed to answer.

Prompt mining tunes the question to the model, not the model to the question.

How it works

You mine candidate phrasings from a corpus, optionally paraphrase to add variety, score each candidate on known facts, and keep the winners — then ask the model with the best ones and combine their votes.

1. Gather facts. Start with subject–object pairs you already know are true (from a knowledge base like Wikidata): (Obama, lawyer), (Paris, France).
2. Mine the corpus. Find sentences containing both the subject and object, then extract the connecting text. Two flavors below produce the candidate templates.
3. Paraphrase. Run the best mined prompts through back-translation (translate out to another language and back) to generate fluent variants the corpus didn't contain.
4. Score. Run each candidate template over a held-out set of known facts and measure how often the model fills the blank correctly.
5. Keep and probe. Retain the top templates (the paper kept T=40 per relation), then query the model and combine the answers across prompts.

The two mining methods

Middle-word mining takes the words sitting between the subject and object. "Barack Obama was born in Hawaii" becomes the template "[X] was born in [Y]." Simple, and it captures the most common phrasing for a relation.

Dependency-based mining parses the sentence and walks the shortest dependency path between subject and object, then spans from the leftmost to rightmost word on that path. This catches templates where the linking words aren't literally in the middle — "The capital of France is Paris" yields "capital of [X] is [Y]" even when modifiers sit between the entities.

Why it works

The gain comes from aligning the prompt with the model's pretraining distribution. Ranked by contribution:

Where it shines

Prompt mining is built for knowledge probing — measuring what factual relations a model has stored. That's its home turf, and the LAMA results are the headline evidence.

• Factual relation extraction: On LAMA's T-REx relations, BERT-base rose from 31.1% (manual prompts) to 39.6% with mined prompts plus an optimized ensemble. BERT-large rose from 32.3% to 43.9%.
• Single-prompt swaps: Even without ensembling, replacing manual with mined-and-paraphrased prompts helped — BERT-base reached 34.1% with a single Man+Para prompt.
• Diagnostics and benchmarking: Because a single hand-written prompt only gives a lower bound on what a model knows, mining produces a tighter, fairer estimate — useful when comparing models or auditing what a model has memorized.

The ceiling is striking: the oracle (counting a fact as known if any mined prompt gets it) hit 52.6% on BERT-base and 56.1% on BERT-large — far above any single prompt, which is exactly why diverse prompts matter.

When to use it (and when not)

Reach for it when:

• You're measuring or extracting factual relations and want the tightest estimate of model knowledge.
• You have a set of ground-truth subject–object pairs to score candidate prompts against.
• A relation is queried at scale, so a one-time mining cost amortizes across many queries.

Skip it when:

• The task isn't relational knowledge probing (creative writing, reasoning, summarization gain nothing here).
• You have no validation facts — without them you can't tell a good mined prompt from a bad one.
• A single hand-written prompt already saturates accuracy on your relation.

Model fit: The original work targeted masked language models — BERT-base and BERT-large — where a relation is a single [MASK] to fill. The core idea (phrasing changes measured knowledge) carries to autoregressive models, but the fill-the-blank scoring is most natural for masked models. For very small models, mined prompts help less because the knowledge often isn't there to elicit.

Escalate when: mining plateaus and you need gradient-level prompt search — move to AutoPrompt, which optimizes discrete trigger tokens by gradient signal rather than corpus frequency.

Structure and components

A complete mining pipeline has these parts:

1. Fact set — known subject–object pairs per relation, split into a set for scoring prompts and a set for final evaluation.
2. Corpus — a large text source (the paper used Wikipedia) where the relation is stated in natural language.
3. Extractor — middle-word and dependency-path logic that turns matched sentences into [X] … [Y] templates.
4. Paraphraser — back-translation with B=7 candidates to diversify phrasings.
5. Scorer — runs each template over the scoring facts and ranks by fill accuracy.
6. Ensembler — combines the kept prompts at query time.

The prompt template

Every mined prompt is just a relation expressed with two slots:

[X] was born in [Y].
capital of [X] is [Y].
[X] worked as a [Y].

[X] is filled with the subject; [Y] is the [MASK] the model predicts.

Core algorithm

The mining-and-ranking loop in sketch form:

def mine_prompts(relation, facts, corpus, top_t=40):
    candidates = {}
    for subj, obj in facts:
        for sent in corpus.sentences_with(subj, obj):
            # middle-word: text between the two entities
            mid = sent.span_between(subj, obj)
            # dependency: leftmost..rightmost on the shortest path
            dep = sent.dependency_span(subj, obj)
            for tmpl in (templatize(mid), templatize(dep)):
                candidates[tmpl] = candidates.get(tmpl, 0) + 1
    # keep the most frequent, then rank by accuracy on held-out facts
    frequent = top_k(candidates, top_t)
    return sorted(frequent, key=lambda t: score(t, facts), reverse=True)

templatize replaces the matched subject and object with the [X] and [Y] slots; score measures how often the template's [MASK] fill matches the true object.

Ensembling

Three ways to combine kept prompts, in rising order of effort and payoff:

• Top-1: just use the single prompt with the best scoring accuracy. Simplest baseline.
• Rank / average: average the model's log-probabilities across the top-K prompts, then take the highest. Cancels per-prompt noise.
• Optimized: learn a weight per prompt (softmax-parameterized) on the scoring set, so reliably-good prompts count more. This is what reached 39.6% on BERT-base and 43.9% on BERT-large.

Configuration

Implementation workflow

1. Pick relations and pull facts. From a KB, gather subject–object pairs per relation; reserve a scoring split and an evaluation split.
2. Mine. Match facts against the corpus, extract middle-word and dependency templates, count frequencies, keep the top T.
3. Paraphrase. Back-translate the strongest seeds to add fluent variants.
4. Score and rank. Evaluate every candidate on the scoring split; sort by fill accuracy.
5. Choose an ensemble. Start with top-1, then try rank-averaging, then the optimized weights if accuracy still matters.
6. Evaluate honestly. Report on the held-out split, and report the oracle so you know your headroom.

Do and don't

Do:

• Hold out the facts used to evaluate from the facts used to score prompts — otherwise you're grading on the training set.
• Keep several prompts per relation; ensembling is where most of the gain past a single mined prompt comes from.
• Report the oracle ceiling so a reader sees how much the phrasing alone is leaving on the table.

Don't:

• Don't mine and score on the same facts.
• Don't assume a mined prompt transfers across models — re-score for each model, since "best phrasing" is model-specific.
• Don't expect mining to conjure knowledge the model never stored; it elicits, it doesn't teach.

Debugging

Testing and how to prove it

The proof is simple: measure micro-averaged fill accuracy on held-out facts, compared against a manual-prompt baseline and the oracle.

def evaluate(prompts, eval_facts, model, ensemble="rank"):
    correct = 0
    for subj, obj in eval_facts:
        if ensemble == "top1":
            pred = model.fill(prompts[0].format(X=subj))
        else:  # rank: average log-probs across prompts
            pred = model.fill_ensemble(
                [p.format(X=subj) for p in prompts]
            )
        correct += (pred == obj)
    return correct / len(eval_facts)

# Oracle: known if ANY prompt gets it (your accuracy ceiling)
def oracle(prompts, eval_facts, model):
    hit = sum(
        any(model.fill(p.format(X=s)) == o for p in prompts)
        for s, o in eval_facts
    )
    return hit / len(eval_facts)

Track three numbers: the manual baseline, your mined-plus-ensemble accuracy, and the oracle. The gap between your result and the oracle is the headroom diverse prompting could still recover.

Limitations

• Probing-only scope. This is a knowledge-probing technique. It doesn't improve reasoning, generation, or any task without a fill-the-blank fact structure.
• Needs ground truth. Without known fact pairs, you can't rank candidates — mining degenerates to guessing.
• Corpus dependence. A relation rarely stated in your corpus yields few, weak templates. Paraphrasing helps but can't fully cover a gap.
• Single-token bias. The original setup predicts a single [MASK], so multi-token objects need extra handling.
• Model-specific results. Best prompts for BERT-base aren't best for BERT-large; re-score per model.
• Elicits, never teaches. If the oracle is low, the knowledge isn't there and no prompt will find it.

Advanced techniques

• Mining + paraphrasing together. The strongest single-prompt configurations combined both sources; mining supplies grounded phrasings, paraphrasing supplies fluent variety.
• Optimized ensembles over top-1. Learning per-prompt weights on the scoring set consistently beat just picking the single best prompt — the bulk of the 31.1% → 39.6% lift.
• Relation-specific T. Some relations saturate with a handful of prompts; others keep gaining. Tune how many prompts you keep per relation rather than fixing one T everywhere.

Ecosystem and integration

Prompt mining sits in the automatic prompt generation family alongside gradient-based search and continuous (soft) prompts. Its artifact — the LM Prompt And Query Archive (LPAQA), released with the paper — is a reusable set of mined and paraphrased prompts per relation.

Hybrids: Mining pairs naturally with paraphrasing (the paper's own combination) and can seed a gradient-based search with strong starting templates. For knowledge-base construction, mined prompts feed a probing pipeline that turns a model into a queryable fact source.

Transition: Start manual, mine once you have ground-truth pairs and a corpus, then escalate to AutoPrompt or soft prompts if you have gradients and need to squeeze elicitation further.

Future directions

• Beyond masked models. Adapting frequency-based mining to autoregressive and instruction-tuned models, where "fill the blank" becomes "complete the sentence."
• Multi-token and multi-hop relations. Extending the single-[MASK] setup to objects that span several tokens or relations that chain.
• Continual prompt archives. Growing the prompt set as new corpora arrive, so the archive tracks how phrasing — and measured knowledge — shifts over model generations.

Summary

• Prompt mining discovers prompt templates by harvesting how a corpus phrases a relation, instead of hand-writing them.
• Two mining methods — middle-word and dependency-path — produce [X] … [Y] templates; back-translation paraphrasing adds variety.
• Candidates are scored on known facts, the top prompts are kept (T=40), and answers are ensembled (top-1, rank, or optimized weights).
• On LAMA, this lifted BERT-base from 31.1% to 39.6% and BERT-large from 32.3% to 43.9%, with oracle ceilings of 52.6% and 56.1%.
• Use it for factual knowledge probing when you have ground-truth pairs and a corpus; skip it for reasoning, generation, or when you lack facts to score against.
• It elicits stored knowledge by matching the model's preferred phrasing — it never teaches the model anything new.

Sources:

• How Can We Know What Language Models Know? (Jiang et al., 2020, TACL)
• arXiv:1911.12543
• LPAQA — LM Prompt And Query Archive (code & prompts)
• AutoPrompt (Shin et al., 2020)