All tutorials
Track 21·State & Control

Your first @evaluator and seek()

GoalSeeking runs a plan-execute-evaluate loop. Mark one method @evaluator, return GoalEvaluation, and call seek() instead of run(). The loop continues until the evaluator says the goal is met.

intermediate10 min
Video coming soon
Browse this tutorial's folder in tutorials-pygithub.com/OpenSymbolicAI/tutorials-py/tree/main/21-goal-seeking

Before you start

run() generates a plan and executes it once. GoalSeeking adds a loop: plan, execute, evaluate. If the evaluator says the goal is not met, the loop runs again with the feedback fed back to the model.

The task#

The agent must find a secret number between 1 and 1,000. After each guess it receives a temperature hint and a direction: "cold — go higher", "hot — go lower", "correct", and so on. Each iteration the model picks the midpoint of the remaining range, guesses it, and the evaluator checks if the hint came back as "correct".

The agent#

python
# guesser.py
from opensymbolicai.blueprints import GoalSeeking
from opensymbolicai.core import decomposition, evaluator, primitive
from opensymbolicai.models import ExecutionResult, GoalContext, GoalEvaluation, GoalSeekingConfig

SECRET = 742
GOAL = "Guess the secret number between 1 and 1000."


def _hint(n: int) -> str:
    diff = SECRET - n
    distance = abs(diff)
    direction = "go higher" if diff > 0 else "go lower"
    if distance == 0:   return "correct"
    if distance < 25:   return f"burning — {direction}"
    if distance < 100:  return f"hot — {direction}"
    if distance < 200:  return f"warm — {direction}"
    if distance < 400:  return f"cold — {direction}"
    return f"freezing — {direction}"


class HintContext(GoalContext):
    low: int = 1
    high: int = 1000
    last_hint: str = "no guess yet"


class Guesser(GoalSeeking):
    """An agent that converges on a hidden number using binary search."""

    def __init__(self, **kwargs) -> None:
        cfg = kwargs.pop("config", None) or GoalSeekingConfig(max_iterations=20)
        super().__init__(config=cfg, **kwargs)

    def create_context(self, goal: str) -> HintContext:
        return HintContext(goal=goal)

    def update_context(self, context: HintContext, execution_result: ExecutionResult) -> None:
        for step in execution_result.trace.steps:
            if step.primitive_called == "guess" and step.success:
                n_arg = step.args.get("n") or step.args.get("arg0")
                if n_arg is None:
                    continue
                n = int(n_arg.resolved_value)
                hint = str(step.result_value)
                context.last_hint = hint
                if "go higher" in hint:
                    context.low = max(context.low, n + 1)
                elif "go lower" in hint:
                    context.high = min(context.high, n - 1)

    @primitive(read_only=True)
    def midpoint(self, low: int, high: int) -> int:
        """Return the midpoint of [low, high]."""
        return (low + high) // 2

    @primitive(read_only=True)
    def guess(self, n: int) -> str:
        """Guess n. Returns a temperature hint and direction, e.g. 'hot — go lower'."""
        return _hint(n)

    @decomposition(
        intent="guess the midpoint of [low, high] from context",
        expanded_intent=(
            "Each iteration the context shows updated low and high values. "
            "Substitute those exact integers into midpoint(), then call guess(n)."
        ),
    )
    def _example(self) -> str:
        # context: low=501, high=1000 after a 'cold — go higher' on 500
        n = self.midpoint(501, 1000)
        result = self.guess(n)
        return result

    @evaluator
    def _check(self, goal: str, context: HintContext) -> GoalEvaluation:
        return GoalEvaluation(goal_achieved=context.last_hint == "correct")

Run it#

python
# main.py
from guesser import GOAL, Guesser
from opensymbolicai.llm import LLMConfig

llm = LLMConfig(provider="ollama", model="qwen2.5-coder:7b")
agent = Guesser(llm=llm)
result = agent.seek(GOAL)

for iteration in result.iterations:
    step = next(
        (s for s in iteration.execution_result.trace.steps if s.primitive_called == "guess"),
        None,
    )
    n = int((step.args.get("n") or step.args.get("arg0")).resolved_value) if step else "?"
    hint = str(step.result_value) if step else "?"
    print(f"iteration {iteration.iteration_number:2d}: guess={n:>4}  hint={hint}")

print()
print(f"status:     {result.status.value}")
print(f"iterations: {result.iteration_count}")
bash
uv run main.py
text
iteration  1: guess= 500  hint=cold — go higher
iteration  2: guess= 750  hint=hot — go lower
iteration  3: guess= 625  hint=warm — go higher
iteration  4: guess= 687  hint=warm — go higher
iteration  5: guess= 718  hint=hot — go higher
iteration  6: guess= 734  hint=burning — go higher
iteration  7: guess= 742  hint=correct

status:     achieved
iterations: 7

The three new pieces#

@evaluator marks the method that decides if the goal is done. It receives the goal string and the current context, and returns a GoalEvaluation. Set goal_achieved=True to stop the loop. Set it False and the loop continues.

seek(goal) drives the loop. It returns a GoalSeekingResult with status, iteration_count, and a list of iterations each carrying its own plan, trace, and evaluation.

GoalStatus records how the loop ended. achieved means the evaluator returned True. max_iterations means the limit was hit before that happened (default is 10 for GoalSeekingConfig; the Guesser sets it to 20).

Context and update_context#

HintContext is a GoalContext subclass. Its fields (low, high, last_hint) are injected into the prompt as literals each iteration, so the model substitutes the current range into midpoint(low, high) without needing to read the raw trace itself.

update_context is called after each execution. It reads the hint from the trace and narrows the range. The evaluator and planner never touch ExecutionResult directly: that boundary is enforced by design.