Directional stimulus prompting

You can't fine-tune a black-box model like GPT-4 or Claude. So how do you steer it on your task without a fortune in API calls or a giant prompt? Directional stimulus prompting (DSP) trains a tiny model to do the steering for you: a small policy network reads each input and writes a short hint, you drop that hint into the prompt, and the frozen big model follows it. Li et al. introduced DSP at NeurIPS 2023, and with just 80 training dialogues it lifted ChatGPT's combined score on MultiWOZ by 41.4%, matching fully supervised systems trained on the whole dataset.

See it work

Say you want a summary that hits the facts a human editor cares about. Plain prompting gives the model nothing to aim at:

Prompt:  Summarize this article in 2-3 sentences.
         [article about a flight delay and refund policy change]

Output:  An airline experienced delays and discussed changes to
         its policies. The situation affected travelers.

Vague, and it misses what matters. Now a small policy model reads the same article and emits a few keywords as a "stimulus," which you splice into the prompt:

Hint (from policy model):  Hint: Delta; 3-hour delay; $200 vouchers;
                           refund policy; December 12

Prompt:  Summarize this article in 2-3 sentences.
         Hint: Delta; 3-hour delay; $200 vouchers; refund policy;
         December 12
         [same article]

Output:  Delta's December 12 flights saw delays of up to three hours,
         prompting $200 vouchers for affected passengers. The airline
         also announced a revised refund policy for future delays.

Same frozen model, same article. The hint pointed it at the entities a good summary needs, and the output snapped into focus. Nobody touched the big model's weights.

The mental model

DSP is a caddie for a golfer you're not allowed to coach. You can't change the player's swing, but you can hand them the right club and read the green before every shot.

The frozen LLM is the powerful-but-generic player. The policy model is the caddie who has studied your course (your task and data) and whispers a tailored tip for each hole (each input). The tip is small and cheap, but it's instance-specific, so it moves the player toward your idea of a good outcome shot after shot.

The flip from ordinary prompt engineering: instead of you hand-writing one prompt and hoping it generalizes, a trainable model writes a fresh, optimized hint for every single input.

How it works

1. The policy model reads the input. A small, tunable language model (the paper uses T5-base at 220M parameters and Flan-T5-large at 750M) takes the raw input and generates a short stimulus: a handful of discrete tokens like keywords for a summary, or dialogue acts for a chatbot turn.
2. You build the prompt. Concatenate your normal instruction, the generated stimulus, and the input. The stimulus is just text, so this works on any API.
3. The frozen LLM generates. The big model never changes. It reads the hint and conditions its output on it, the same way it would condition on any in-context instruction.
4. You score the output and train the policy. Compare the LLM's output to a reference and turn that into a reward. The policy model learns to emit stimuli that make the frozen LLM produce high-reward outputs.

Training has two stages. First, supervised fine-tuning (SFT): build target stimuli from labeled data (for summarization, extract keywords from each reference summary with TextRank and keep the ones that appear in it) and teach the policy to predict them. Then, reinforcement learning: optimize the policy with PPO, using the frozen LLM's downstream score as the reward (ROUGE-Avg against the reference for summarization, the task's combined score for dialogue). The RL stage is what closes the loop, because it directly rewards hints that the specific frozen model responds well to.

Why it works

Where it shines

DSP fits any task where you have a notion of a "good" output you can score, plus a frozen model you can't fine-tune. The paper evaluates three:

• Summarization (CNN/Daily Mail). The stimulus is keywords the summary should cover. Trained on as few as 1,000-4,000 article-summary pairs, DSP lifted ChatGPT's ROUGE, BLEU, and Meteor scores by 1-2 points over standard prompting.
• Dialogue response generation (MultiWOZ). The stimulus is dialogue acts and slot-value pairs. This is the headline result.
• Chain-of-thought reasoning. Instance-specific CoT stimuli beat human-written reasoning prompts for InstructGPT.

The MultiWOZ numbers (2.0, trained on 80 dialogues = 1% of the data) show how far a tiny steering model can move a frozen LLM:

The combined score ((Inform + Success) / 2 + BLEU) jumped from 68.4 to 96.7 for ChatGPT, the 41.4% relative gain that matches or surpasses fully supervised models trained on all 8,438 dialogues. Notice BLEU barely moves: the win comes from the policy steering the model toward the right intents and slots (inform and success), not from copying reference phrasing.

When to use it (and when not)

Reach for DSP when:

• The strong model is a frozen API you cannot fine-tune.
• You run the task at high volume, so amortizing a one-time training cost pays off.
• You have labeled data and a metric to score outputs (ROUGE, task success, accuracy).
• A static prompt plateaus and you need per-input guidance.

Skip it when:

• You can just fine-tune the target model, or a hand-tuned few-shot prompt already clears your bar.
• The task is one-off or low-volume; training a policy isn't worth the setup.
• You have no reference outputs or no usable reward signal.

Model fit. DSP needs a capable frozen LLM that actually follows in-context hints; the paper used Codex (code-davinci-002), ChatGPT (gpt-3.5-turbo), and InstructGPT (text-davinci-002). The policy can stay small (220M-750M) because it only writes hints, not full answers.

Escalation. If a fixed prompt and few-shot examples don't get you there and you can't fine-tune the model, DSP is the next step up. If you can fine-tune the target model and have plenty of data, full fine-tuning usually wins outright.

Components and structure

A DSP system has three pieces:

• The policy model. A small tunable LM (T5 / Flan-T5) that maps input to stimulus.
• The stimulus. Short discrete tokens, shaped to the task: keywords for summaries, dialogue acts and slot-value pairs for chatbots, reasoning cues for CoT.
• The frozen LLM. The untouched black-box generator.

The prompt you actually send follows a fixed template:

[task instruction]
Hint: [stimulus tokens from the policy model]
[input]

The training loop

# Stage 1: supervised fine-tuning of the policy model.
# Targets are stimuli built from labels (e.g., TextRank keywords
# kept only if they appear in the reference summary).
for x, target_stimulus in labeled_data:
    loss = cross_entropy(policy(x), target_stimulus)
    loss.backward(); optimizer.step()

# Stage 2: reinforcement learning (PPO) against the frozen LLM.
for x, reference in rl_data:
    stimulus = policy.sample(x)                # explore hints
    prompt   = build_prompt(instruction, stimulus, x)
    y        = frozen_llm(prompt)              # black-box call
    reward   = score(y, reference) * 10        # e.g., ROUGE-Avg, rescaled
    policy.ppo_update(x, stimulus, reward)     # reward good hints

Inference is trivial once trained: run the policy, splice the hint, call the LLM.

def answer(x):
    stimulus = policy.generate(x)
    prompt   = build_prompt(instruction, stimulus, x)
    return frozen_llm(prompt)

Configuration

Do and don't

• Do start with SFT so RL has a sane policy to explore from; pure RL from scratch is slow and unstable.
• Do pick a reward that is your real objective, not a loose proxy.
• Don't make the stimulus long; a few focused tokens steer better than a paragraph.
• Don't assume a policy trained against one frozen model transfers to another; the reward is model-specific.

Debugging

• Hints ignored by the LLM -> the frozen model may not follow that hint format; try a clearer template or a stronger LLM.
• Policy collapses to generic hints -> reward is too flat; sharpen or rescale it.
• Great SFT, weak after RL -> PPO is over-exploring; lower the learning rate or add a KL penalty toward the SFT policy.
• Metric up, quality down -> your reward is gameable; the model is hitting the score without real improvement.

How to prove it works

Hold out a test split the policy never saw, then compare three arms on the same frozen LLM: standard prompting, DSP after SFT only, and DSP after SFT+RL. The paper's MultiWOZ table is exactly this ablation, and it shows SFT alone already helps (66.8 to 81.7 for Codex) while RL delivers the rest (to 93.3). Report the metric you trained against plus a couple you didn't, so you catch reward gaming. Watch for a metric climbing while a held-out human or secondary score stalls.

Limitations

• Training is expensive and fiddly. Tens of thousands of reward-bearing LLM calls plus PPO tuning is a real engineering lift, far heavier than writing a prompt.
• You need labels and a reward. No reference outputs, no scoreable metric, no DSP.
• It's task- and model-bound. Retrain the policy when the task, the metric, or the frozen LLM changes.
• Reward hacking. Optimizing a proxy metric can inflate the score without improving real quality.
• The ceiling is the frozen model. Hints redirect attention; they can't add knowledge the big model lacks.

Ecosystem and related techniques

DSP sits among black-box adaptation methods. Unlike soft-prompt or prefix tuning, it needs no gradients through the LLM, so it works on closed APIs. It pairs naturally with retrieval (let the policy condition on retrieved context) and with chain-of-thought (the stimulus can be a reasoning cue). The implementation builds on Hugging Face Transformers and an RL library for PPO; the authors released code on GitHub.

Conceptually it's a cousin of automatic prompt optimization (APE, RLPrompt) but with a twist: instead of searching for one good prompt, it trains a model to emit a new hint per input. Where few-shot prompting hands the model fixed examples, DSP hands it a dynamic, learned pointer.

Future directions

Open questions include making the RL loop cheaper (fewer reward-bearing LLM calls), letting one policy steer multiple frozen models, and extending stimuli beyond keywords and dialogue acts into richer plans for agents and tool use. As black-box APIs stay the norm, training small controllers to steer them is a durable pattern.

Summary

• DSP trains a small policy model to write an instance-specific hint that steers a frozen black-box LLM, no fine-tuning of the big model required.
• The policy is tiny (T5-base 220M, Flan-T5-large 750M) and learns in two stages: supervised fine-tuning on distilled labels, then PPO against the LLM's downstream score.
• Stimuli are short and task-shaped: keywords for summaries, dialogue acts for chatbots, reasoning cues for CoT.
• Headline result: 80 dialogues lifted ChatGPT's MultiWOZ combined score from 68.4 to 96.7 (+41.4%), matching fully supervised models trained on 8,438 dialogues; summarization gained 1-2 points in ROUGE, BLEU, and Meteor.
• Reach for it when the strong model is frozen, the task is high-volume, and you have labels plus a metric to reward; skip it for one-off tasks or when you can just fine-tune.
• Cost is front-loaded in training (tens of thousands of reward-bearing LLM calls); inference is cheap.
• Source: Li, Peng, He, Galley, Gao, and Yan, "Guiding Large Language Models via Directional Stimulus Prompting," NeurIPS 2023 (arXiv:2302.11520).