Reflexion: a complete guide

Most models get one shot at a task, fail, and never learn why. Reflexion adds a feedback loop: the model attempts the task, reads what went wrong, writes itself a short note about how to do better, and tries again. That verbal "lesson" is the trick. On the HumanEval coding benchmark, Reflexion hit 91% pass@1, beating the GPT-4 baseline of 80% reported in the same paper (Shinn et al., 2023).

See it work

Take a coding task where the first attempt has a subtle bug. A normal one-shot run stops here:

Task: Write is_palindrome(s) that ignores case and spaces.

Attempt 1:
def is_palindrome(s):
    return s == s[::-1]

Unit test result: FAILED on "Race car" (case + spaces not handled).

A plain retry might just shuffle the same mistake. Reflexion instead pauses to write a lesson, then retries with that lesson in context:

Self-reflection: My function compared the raw string. I ignored the
spec's "ignore case and spaces." Next time, normalize first: lowercase
and strip non-alphanumeric characters before comparing.

Attempt 2 (with reflection in memory):
def is_palindrome(s):
    cleaned = [c.lower() for c in s if c.isalnum()]
    return cleaned == cleaned[::-1]

Unit test result: PASSED.

Same model, same task. The only new ingredient is the model's own written critique, fed back in before the second try.

The mental model

Think of a student doing practice problems with an answer key. They try a problem, check the result, and scribble a margin note: "I keep forgetting to carry the one." Next problem, that note is right there in front of them. They didn't get smarter overnight, they just stopped repeating one specific mistake.

Reflexion is reinforcement learning done in words, not weights. The "policy update" is a sentence the model writes to itself.

That's the core flip. Classic reinforcement learning nudges millions of parameters with gradients. Reflexion freezes the model and stores the lesson as plain text in memory, which is cheaper, faster, and you can actually read what was learned.

How it works

Reflexion runs three roles in a loop. An Actor does the task, an Evaluator scores the attempt, and a Self-Reflection model turns that score into a written lesson stored in memory. The next attempt sees the lesson and tries again.

Step by step:

1. Act. The Actor generates actions or text conditioned on the current state. It can be a plain LLM, or wrap Chain-of-Thought or ReAct underneath.
2. Evaluate. The Evaluator produces a reward signal for the attempt. This can be an exact-match check for reasoning, a heuristic or environment signal for decision-making, or self-generated unit tests for code.
3. Reflect. The Self-Reflection model reads the trajectory and the sparse reward, then writes specific, natural-language feedback about what to change. This turns a binary "you failed" into "you failed because X, so do Y."
4. Remember. The reflection is appended to an episodic memory buffer.
5. Retry. The Actor runs again with the reflections in context, repeating until it succeeds or hits a trial limit.

Why it works

Where it shines

The paper tests three task families, and the gains hold across all of them:

• Coding. 91% pass@1 on HumanEval (Python), above the GPT-4 baseline of 80%. On MBPP the picture is more nuanced: Reflexion scored 77.1% pass@1, just under the GPT-4 baseline of 80.1%, because self-generated tests don't always capture the spec.
• Sequential decision-making. On ALFWorld (text-based household tasks), ReAct + Reflexion solved 130 of 134 tasks over 12 trials, roughly a 22% absolute jump over plain ReAct, which plateaued.
• Multi-hop reasoning. On 100 HotPotQA questions, CoT with ground-truth context plus Reflexion rose from 54% to 68% accuracy, a 14-point gain; ReAct + Reflexion improved by about 20%.

The pattern: Reflexion helps most when a task has a clear pass/fail signal and a multi-step path that's easy to get subtly wrong.

When to use it (and when not)

Reach for it when:

• You have a real feedback signal: unit tests, an environment reward, an exact-match grader, or a reliable judge.
• The task is multi-step and the model fails in ways it could diagnose if it looked back.
• You can afford a few extra attempts per task in exchange for accuracy.

Skip it when:

• There's no way to score an attempt, since reflection needs something to react to.
• The task already succeeds in one shot, so retries just burn tokens.
• Latency or cost per task is tight and you can't afford multiple trials.

Model fit. Self-correction is an emergent ability of stronger, larger models. The authors found a weaker model (StarChat-beta) scored 0.26 pass@1 with and without Reflexion, no gain at all, because it couldn't write useful reflections. Use a capable base model.

Escalation. If single-pass prompting underperforms but you have a grader, add Reflexion. If reflections stop helping after a couple of trials, the bottleneck is usually the base model or the evaluator, not the loop.

Components and structure

Reflexion has three model roles plus a memory store:

• Actor. Generates the trajectory (actions and text). Often built on Chain-of-Thought or ReAct.
• Evaluator. Maps a trajectory to a reward. Exact-match for reasoning, heuristics or environment signals for decision-making, self-generated unit tests for code.
• Self-Reflection model. An LLM that converts the trajectory plus sparse reward into verbal feedback.
• Memory. Short-term is the current trajectory; long-term is an episodic buffer of past reflections.

The core loop

In pseudocode, the whole technique fits in a few lines:

memory = []                      # episodic buffer of reflections
for trial in range(max_trials):
    trajectory = actor.run(task, reflections=memory)
    reward, feedback = evaluator.score(trajectory)
    if reward.is_success():
        return trajectory
    reflection = self_reflect(task, trajectory, feedback)
    memory.append(reflection)     # grow the lesson list
    memory = memory[-OMEGA:]      # keep only the last Omega lessons
return trajectory                 # best effort after max_trials

The reflection prompt is the heart of it. Keep it focused on diagnosis, not a rewrite:

You attempted the task and failed. Here is your trajectory and the
evaluation result:

{trajectory}
{evaluation_feedback}

In a few sentences, diagnose what went wrong and state a concrete
strategy to do better next time. Do not solve the task now.

Configuration

Implementation workflow

1. Define a reliable success signal: tests, a grader, or an environment reward.
2. Pick the Actor base (ReAct for agents, CoT for reasoning, plain LLM for generation).
3. Wire the loop: act, evaluate, reflect, append, retry.
4. Set a trial cap and a memory window (Omega).
5. Log every reflection so you can read what the model is learning.
6. Tune the reflection prompt until critiques are specific and actionable.

Do:

• Make the evaluator trustworthy first; a noisy signal teaches the wrong lesson.
• Keep reflections short and diagnostic.
• Stop early on success to save tokens.

Don't:

• Let reflections rewrite the answer; they should guide the next attempt, not be it.
• Stuff unlimited reflections into context; trim to the window.
• Use a weak base model and expect self-correction to appear.

Debugging

Testing and proving it works

Compare Reflexion against a single-pass baseline on the same tasks, scored by the same evaluator. Track success rate by trial number; a working setup should climb across trials and then flatten. For code, the authors generate unit tests via Chain-of-Thought, filter them for syntactic validity with AST parsing, and sample up to six tests per problem. Watch for tests that pass for the wrong reason, since a bad grader makes reflections actively misleading.

Limitations

• Local minima. Reflexion is verbal policy optimization, so it can get stuck repeating a flawed strategy it can't reason its way out of.
• Self-evaluation dependence. The loop is only as good as the model's ability to critique itself; weaker models gain nothing (StarChat-beta stayed at 0.26 pass@1).
• Simple memory. Long-term memory is just a sliding window. The authors flag richer stores (vector databases, SQL) as future work.
• Code-specific gaps. Test-driven evaluation struggles with non-deterministic generators and impure functions that hit external APIs, so the test signal can be unreliable.
• Cost. Multiple trials per task multiply latency and token spend.

Ecosystem and related techniques

Reflexion sits in the self-improvement family. Self-Refine critiques and revises within a single session but keeps no memory across attempts. Self-consistency samples many answers and votes, with no critique at all. Chain-of-Thought and ReAct are the usual Actors inside Reflexion rather than competitors. The natural pairing is ReAct (for acting in an environment) plus Reflexion (for learning between episodes), which is exactly the ALFWorld setup.

Future directions

The authors point to richer memory (vector or relational stores instead of a flat window), better evaluators so the reward signal is trustworthy on messier tasks, and combining verbal reflection with other feedback sources. The broader idea, that an agent can improve by writing and re-reading its own lessons, underpins much of the later work on long-running, self-improving agents.

Summary

• Reflexion turns a failed attempt into a written lesson the model reads before retrying, which is reinforcement learning in words, not weights.
• Three roles drive it: an Actor that attempts, an Evaluator that scores, and a Self-Reflection model that writes the lesson into episodic memory.
• Headline results (Shinn et al., 2023): 91% pass@1 on HumanEval vs 80% baseline, 130/134 ALFWorld tasks over 12 trials, and 54% to 68% on HotPotQA with ground-truth context.
• Use it when you have a real pass/fail signal, a multi-step task with fixable errors, and a strong base model; skip it without a grader, on one-shot-easy tasks, or with weak models.
• The costs are real: multiple trials per task, dependence on self-evaluation quality, and a memory that's just a small sliding window.