← All tutorials

Track 22 · Data & Types

Intermediate data lives in Python, not the prompt

When one primitive returns a list and another consumes it, the data travels as a Python variable. The model never sees it. This holds whether the list has 100 entries or 100,000.

Intermediate8 min
Video coming soon
Project codeBrowse this tutorial's folder in tutorials-pygithub.com/OpenSymbolicAI/tutorials-py/tree/main/22-pipe

When one primitive produces a large list and another consumes it, the list travels between them as a Python variable in the plan's namespace. The model never sees those values: not when the list is produced and not when it is consumed. This tutorial shows that in action with 100 entries and 100,000.

The agent: ReturnAnalyst

The agent holds a series of synthetic monthly returns and exposes arithmetic over them as @primitive methods.

python
# analyst.py
import random
import statistics
from opensymbolicai.blueprints import PlanExecute
from opensymbolicai.core import primitive


def generate_monthly_returns(n: int, seed: int = 42) -> list[float]:
    rng = random.Random(seed)
    return [round(rng.gauss(0.008, 0.04), 6) for _ in range(n)]


class ReturnAnalyst(PlanExecute):
    def __init__(self, n: int, **kwargs) -> None:
        super().__init__(**kwargs)
        self._monthly = generate_monthly_returns(n)

    @primitive(read_only=True)
    def monthly_returns(self) -> list[float]:
        """Monthly return series for the full dataset."""
        return self._monthly

    @primitive(read_only=True)
    def mean_of(self, values: list[float]) -> float:
        """Arithmetic mean of a list of floats."""
        return round(sum(values) / len(values), 6)

    @primitive(read_only=True)
    def count_positive(self, values: list[float]) -> int:
        """Number of positive values in the list."""
        return sum(1 for v in values if v > 0)

    @primitive(read_only=True)
    def len_of(self, values: list[float]) -> int:
        """Number of entries in the list."""
        return len(values)

    @primitive(read_only=True)
    def max_of(self, values: list[float]) -> float:
        """Largest value in the list."""
        return round(max(values), 6)

    @primitive(read_only=True)
    def min_of(self, values: list[float]) -> float:
        """Smallest value in the list."""
        return round(min(values), 6)

    @primitive(read_only=True)
    def subtract(self, a: float, b: float) -> float:
        """Subtract b from a."""
        return round(a - b, 6)

    @primitive(read_only=True)
    def divide(self, a: float, b: float) -> float:
        """Divide a by b."""
        return round(a / b, 6)

The model sees only the signatures: monthly_returns() -> list[float], mean_of(values: list[float]) -> float, and so on. The data itself is invisible to it.

Run three tasks at two scales

python
# main.py
from analyst import ReturnAnalyst
from opensymbolicai.llm import LLMConfig

TASKS = [
    "What is the mean monthly return?",
    "What fraction of months had a positive return?",
    "What is the range between the best and worst monthly return?",
]

llm = LLMConfig(provider="ollama", model="qwen2.5-coder:7b")

for task in TASKS:
    print(f"\nTask: {task}\n")
    for n in [100, 100_000]:
        agent = ReturnAnalyst(n=n, llm=llm)
        result = agent.run(task)
        print(f"  n={n:>7,}  result = {result.result}")
        print(f"  plan:")
        for line in result.plan.splitlines():
            print(f"    {line}")
        print()
bash
uv run main.py

Output (values vary by run):

text
Task: What is the mean monthly return?

  n=    100  result = 0.010329
  plan:
    returns = monthly_returns()
    mean_return = mean_of(values=returns)
    return mean_return

  n=100,000  result = 0.00809
  plan:
    mean_monthly_return = mean_of(monthly_returns())
    return mean_monthly_return


Task: What fraction of months had a positive return?

  n=    100  result = 0.63
  plan:
    positive_count = count_positive(monthly_returns())
    total_months = len_of(monthly_returns())
    fraction_with_positive_return = divide(positive_count, total_months)
    return fraction_with_positive_return

  n=100,000  result = 0.58112
  plan:
    ...


Task: What is the range between the best and worst monthly return?

  n=    100  result = 0.198228
  plan:
    best_return = max_of(monthly_returns())
    worst_return = min_of(monthly_returns())
    range_between_best_and_worst = subtract(a=best_return, b=worst_return)
    return range_between_best_and_worst

  n=100,000  result = 0.376076
  plan:
    ...

The plan text is the same at n=100 and n=100,000. The model wrote two lines of code. The scale of the data does not change that.

Why the plan doesn't change with data size

The plan is generated once from the signatures. monthly_returns() returns list[float] whether that list has 100 items or 100,000. The model plans against the type, not the data.

When the plan executes, monthly_returns() returns the real list, and mean_of(values=returns) receives it directly as a Python variable. Neither call serializes the list into a string or a message.

text
OSAI -- data as a Python variable
--------------------------------------------------
  monthly_returns()
        |
        |  100,000 floats
        |  (Python namespace -- model never sees this)
        v
  mean_of(values=returns)  -->  result
--------------------------------------------------
  model sees: 2 lines of code, 0 floats

Compare: tool-calling loops

In a ReAct / tool-calling loop, every primitive return becomes a message. The list must travel through the context window before the next tool can be called.

text
Tool-calling loop -- data through the context window
--------------------------------------------------
  monthly_returns()
        |
        v
  context: [0.023, -0.041, 0.012, 0.008, ...]  <- 100,000 floats
        |
        |  model copies them into the next call
        v
  mean_of(values=[0.023, -0.041, 0.012, ...])  <- 100,000 floats again
--------------------------------------------------
  model sees: 100,000 floats x 2

At n=100 this is awkward. At n=100,000 it exceeds most context windows. OSAI's plan runs the same code without touching the model either way.

What to notice

  • The plan text is identical at both sizes. Read the two plan outputs for the same task and confirm they say the same thing.
  • Execution time scales with data size; plan time does not. The model call happens once regardless of how many floats are in the list.
  • The pipe is just a variable name. returns = monthly_returns() is ordinary Python assignment. The framework adds no special mechanism here: it runs the plan in a namespace, so intermediate values are naturally in scope.