# A defined return value for the playground's answer

## The problem, in one exchange

Asked for a figure, the playground returns a sentence.

    Question: Find Balance Sheet: GAAP book value (per share). Return Q1 2026 value
    Returns:  "GAAP book value per share as of March 31, 2026 (Q1 2026) was $11.87."
    Wanted:   "$11.87"

Reproduced on the live corpus on 2026-07-28 against `kref-q1-26-10-q`. The
same question against `brsp-q126-10q` returns "GAAP book value per share for
Q1 2026 (as of March 31, 2026) was $7.05." The figure is right both times and
the citation resolves to a real cell. The figure is simply not available as a
figure.

The cause is not missing structured output. `agent.py` already runs Pydantic AI
with `output_type=GroundedAnswer`:

    class GroundedAnswer(BaseModel):
        answer: str
        cited_ids: List[str]

`answer` is a free string no matter what was asked. The envelope is typed; the
payload is not. Everything downstream — a caller in the src package, a
spreadsheet cell, a comparison against last quarter — has to parse English to
recover a number the model already held.

## What this decides

Which shape an answer comes back in, and who decides that shape. Three
candidates were built and measured rather than argued:

- **fixed** — one envelope, `Answer`, whose payload is a closed union: scalar,
  series, grid, prose, unanswerable. The model picks the payload, so choosing
  between a figure and an explanation happens inside the call that answers.
- **planned** — a planner reads the question alone and declares the fields the
  answer should have. Those compile into a Pydantic model the answering agent
  is constrained to. The shape differs per question by construction.
- **declared** — the caller states the shape alongside the question and nothing
  is inferred. What a src caller does when it knows what it is asking for.

Scope decision taken before building: only value-seeking questions get a
constrained shape; explanatory questions keep returning prose. That makes
deciding which is which part of the design, and each candidate puts that
decision somewhere different — inside the answering call for **fixed**, in a
separate planner call for **planned**, outside the system for **declared**.

## How it was measured

**The questions.** 64 questions over the 8 REIT documents already ingested in
`ade_playground` — ACR, BRSP, KREF (10-Q and earnings release), LADR, LFT, and
SEVN (10-Q and presentation). 48 value-seeking, evenly split between terse
operator shorthand and natural sentences, 6 per document; plus 16 prose
controls that cannot be answered by one figure.

The FinanceBench gold questions loaded in this database were not usable: they
cover filings that are not ingested.

**The gold values.** One agent per document authored questions by reading the
printed tables and copying a figure character-for-character out of a named
cell. Code then re-read each named cell from the database and compared. 48 of
48 matched; none were dropped, and none were repaired — a benchmark answer
that had to be corrected is not a benchmark answer.

**The comparison.** Retrieval runs once per question and the identical chunks
are handed to all four arms, so a difference in results is a difference in
return shape and not in what was retrieved.

**The scoring is deterministic.** No judge is asked whether two figures agree.
A returned figure is compared to gold as a string and, separately, as a number.
The baseline has no value field, so it is scored on whether the gold figure
appears anywhere in its prose — measuring whether the information was found,
which is the point this comparison has to be fair about.

## What the measurement showed

Over the 47 value questions whose retrieval reached the gold cell, and the 16
prose controls:

| | baseline | fixed | planned | declared |
|---|---|---|---|---|
| figure correct as a number | — | **47/47** | 46/47 | **47/47** |
| figure exact as printed | — | **45/47** | 37/47 | 44/47 |
| returned a value field at all | 0/47 | **47/47** | 46/47 | **47/47** |
| value field free of prose | 0/47 | **47/47** | **46/46** | **47/47** |
| right shape on value questions | 0/47 | **47/47** | **47/47** | **47/47** |
| kept prose on prose controls | 16/16 | 14/16 | 15/16 | 16/16* |
| cited something | 47/47 | 47/47 | 46/47 | 47/47 |
| errors | 0 | 0 | 0 | 0 |
| mean seconds | 3.01 | 3.33 | 9.90 | 3.33 |

\* **declared** scores 16/16 on routing only because the caller was told which
questions were value-seeking. Its routing is perfect by construction, not by
merit, and that column should be read as "not applicable" rather than "best".

**Typing did not make the model find the number more often.** The baseline
already located the correct figure in 46 of 47 answers. This is the finding to
resist overselling: constraining the output changes whether a caller can *use*
the figure without parsing English, not whether the system *knows* it. The one
baseline case scored as a miss is a sign-convention difference — the cash flow
statement prints `(21,914)` for an outflow and the prose wrote "paid $21,914
thousand", carrying the sign in a word. Scoring stayed sign-strict rather than
adopting a lenient rule that could mask genuine sign errors.

**What "exact as printed" misses are.** `fixed`'s two failures both add a
currency sign the cell did not print — `$ 14.90` where the cell prints `14.90`.
Numerically correct, and both are counted as failures above. This argues that
for programmatic callers the contract is `number` plus `unit`, not the `value`
string.

**Why `planned` loses.** Two concrete failures, not a matter of taste:

1. It names the field after the question, so the model returns
   `net_income_loss_attributable_to_common_stockholders_three_months_ended_march_31_2025`
   for one question and `total_assets` for the next. No caller can program
   against a field name that changes per question — which defeats the purpose
   of the exercise, since the target state is a value consumed in code.
2. Declaring a field as a number discards the printed form: `64,673,125` comes
   back as `64673125.0` and `$2,419.6` as `2419.6`. Nine of its ten
   exact-match failures are this, not arithmetic.

It is also three times slower, because it is two model calls.

**Where `fixed` deviated on prose controls.** Both cases returned a `grid`
where the control expected prose, and both are defensible: "Compare the three
loans SEVN originated in Q1 2026 — how do they differ in collateral type,
market, size, pricing spread, and maturity" came back as a four-column table
comparing exactly those attributes. That is arguably a better answer than a
paragraph. Counted as a routing failure above rather than argued away.

## Recommendation

**Adopt `fixed` — one envelope with a closed union payload.**

- It is the only candidate that is literally a single defined return value.
  One type comes back always; the payload varies. A caller writes one
  `isinstance` branch, not a schema negotiation.
- Accuracy ties the best arm (47/47 numeric) at the same latency (3.33s), and
  it is the most accurate on exact printed form (45/47).
- It decides the shape itself. `declared` matches it only when something else
  already knows the answer's shape, which the person typing into the playground
  does not.
- The shapes are written down. A closed union can be reviewed, versioned, and
  exhaustively handled; a schema invented at runtime cannot.

`declared` is not discarded — it stays as the src-facing entry point, and it is
the same call underneath. A caller in code that knows it wants a scalar should
say so and skip the union. `planned` should not be pursued.

## What grounding buys here that prose cannot

Table chunks carry per-cell ids — `<td id="t0-15-1">$2.97</td>` — and the UI
already resolves a cited id to a page and bounding box. A scalar names the
single cell its figure came from in `source_id`, which makes a check available
that no prose answer admits: the returned figure can be compared against the
text printed in the cell it claims to have read, server-side, with no model in
the loop.

This is the strongest argument for a typed value in a financial setting. A
sentence asserts. A scalar with a cell reference can be checked.

## Constraints the shape survives

**Streaming.** Verified on 2026-07-28 against the typed envelope: a prose
question emitted 11 deltas and closed with a validated `prose` payload; a
scalar question emitted 0 deltas and closed with a validated `scalar` payload.
A figure having nothing to animate is correct behaviour, not a regression — the
stream still opens, validates, and delivers a final object.

**A workflow graph, not one shot.** The agent is a single pass today and is
expected to become a graph with feedback loops. `Answer` does not assume one
pass: it is the type of the value that leaves the graph, and a loop that
revises an answer produces another `Answer`. The `unanswerable` payload matters
here — it gives a feedback loop something explicit to branch on instead of
reading English to learn that retrieval came back short.

**The src package is the target.** Nothing in the shapes depends on the
playground: they are Pydantic models over retrieved chunks, and the call that
fills them takes a question and chunks.

## Known limits of this evidence

- 12 of the 48 gold figures appear in more than one cell of their document. A
  right-looking answer read off the wrong row would still score correct on
  those.
- 1 of 48 questions had retrieval miss the gold cell; it is excluded from the
  47-question scoring above and is a retrieval matter, not a shape matter.
- The gold questions were authored by agents reading the same extracted tables
  the system answers from. This measures the return shape, which is what it was
  built to measure. It does not independently validate extraction.
- `claude-opus-4-8` ignores the `temperature=0` that `agent.py`, `selection.py`
  and all three candidates pass; pydantic-ai warns and drops it. The runs are
  therefore not strictly deterministic. Pre-existing, and outside this ticket.

## Deliberate non-goals

- No change to retrieval. It runs once per question and every arm sees the same
  chunks.
- No new persistence. Provenance stays in-workflow.
- No attempt to type every answer. Explanatory questions keep prose.