Playground RAG, 6 KREF filings, 558 sweep questions

KREF Context Trials

Which differences between the cheaper contexts and the full 10-chunk context are material, and which are noise. Findings from the runs of 8 and 9 September 2026.

Answer model: Opus 5Selection agent: Haiku 4.5, onQuestions: 93 per filing, expectation = valueComparison: printed figures matched across thousands/millions scale

What was compared

Every run used the same six filings, the same 93 questions per filing, the same retrieval window and selection agent, and the same Opus answer schema. Only what Opus was handed changed.

Run	What Opus receives	Opus input, share of E
E the reference	All 10 chunks, as the playground builds them today, plus one line naming the document, its form type and its reporting period.	100%
C	The agent's picks plus the next 3 chunks from the similarity ranking, plus the same document line. Typically 4 to 6 chunks instead of 10.	60%
D	C, and for each picked table a second Haiku call names the row that holds the figure. Opus gets that row instead of the whole table when the call succeeds.	58%

E is the right reference because the document line is a fix in its own right: without it, Opus refused to date charts on the supplement pages. Comparing C and D against E isolates the effect of the smaller context.

The noise floor

Before reading any difference as real, note what happens when nothing changes. E was run against the exact same contexts the earlier 10-chunk run used. Same chunks, same model, same schema, one prompt line added.

Answers changed

of 558, with identical context

Gained

6 of them the supplement's charts, now dated

Lost or changed

run-to-run variation

So a swing of ten or so answers between two runs is within what the model does on its own. Only the individual differences tell you whether a context is worse.

Headline counts

Against E	Same figure	Different figure	E answered, this run declined	This run answered, E declined	Net answers
C	254	2	13	11	−2
D	254	2	13	12	−1

Both cheaper contexts land inside the noise floor on counts alone. The material question is what is inside those 26 and 27 differences.

Material differences: C

I read every one of the 26 against the cited cells and the model's stated reason. Three kinds appear.

Wrong figure 4 answers C got wrong that E did not.

Real estate debt, FY 2025 10-K. C returned $5,145,832, the commercial real estate loans line. That is an asset, not debt. E declined.
Unfunded commitments, Q4 2025 supplement. C returned $454.4, which is $419 of future funding plus the $35.4 CMBS investment outstanding. The unfunded part of that investment is $4.6, not $35.4. The figure appears nowhere in the document and the citation points at a cell reading $419.
Credit and spread marks, Q4 2025 release. C returned 74%, the share of financing that is not marked to market at all. The question asks for the opposite quantity.
Weighted average LTV on new originations, Q1 2026 release. C returned 66%, the whole portfolio's LTV from the highlights text. The originations table prints 72% for the quarter's loans. E had 72%.

Real answer lost 10 figures the document prints, which E returned and C declined. Two causes.

Label strictness, 6. With fewer chunks in view Opus refused synonyms it accepted at 10: "term loans" against Secured term loan, net; "regular dividend" against Dividend per Share; "WA term to maturity" against Max Remaining Term; "first mortgages" against Senior Loans three times, because the question mentions Note 3 and a release has no notes.
Source chunk not in the smaller context, 4. Preferred equity on the 10-Q, term loans on the Q4 supplement where the agent picked portfolio tables rather than the balance sheet, and two Q1 supplement sums where one component fell outside the picks.

Not material 12 differences that are not errors.

7 answers C gave that are correct and E declined out of caution: five property-type percentages read off the 10-Q's portfolio chart, the Q1 supplement's senior loan total for first mortgages, and the single 4-rated loan's principal as the 4-rated bucket.
3 answers E gave that were themselves shaky, so C's decline is the better outcome: the CECL allowance offered as "credit and spread marks", a 0% offered for a dollar question, and a one-component answer to a question that asks for a sum.
2 answers where the two runs read different printed rows for the same measure, 65% against 66% for portfolio LTV, and a first-mortgage total that includes or excludes one held-for-sale loan.

Material differences: D

D shares 17 of its 27 differences with C, including the LTV and credit-mark errors and most of the label-strictness losses. What the row resolver changed on its own:

Wrong figure 3 wrong answers, 1 new. Florida at 3.2% for the loan portfolio's geography, read from a row of the CMBS trust geography table in Note 8. Right number, wrong table. The other two, LTV 66% and credit marks 74%, are the same as C.

Real answer lost 8 printed figures lost. Six are C's label and context losses. Two are new: the 10-Q first-mortgage total, declined because Note 3 prints thousands and the question says millions, and the Q1 supplement's other earning assets sum, declined because one component was not in the row it was handed.

Not material D's gains that C did not have are sound: the origination LTV cited to its exact cell, the Q4 5-rated bucket summed from the four rated loans to $572.2, which matches the $572,191 the 10-K prints for the same date, and the Q4 future funding figure of $419 as printed.

What the resolver did. It was asked about 992 picked tables. It named a row on 223 and declined on 765. Most declines are correct, because most picked tables belong to questions the document does not answer. The effect on Opus input was 2 percentage points below C. Each resolver call costs about a third of a cent on Haiku, so the second pass roughly breaks even on this workload.

What is material, in plain terms

	Wrong figures introduced	Printed figures lost	Opus input
E, 10 chunks, document line	0	0	100%
C, picks plus 3	4	10	60%
D, C plus row resolver	3	8	58%

Cutting the context to 60 percent costs about 4 wrong numbers and 10 missed numbers in 558 questions, roughly one question in forty. The wrong numbers matter more than the misses: each one is a real figure from the document with a real citation, placed under the wrong label, and nothing downstream can tell.

The row resolver does not earn its place. It removes one wrong figure and two misses relative to C, adds one wrong figure and two misses of its own, and cuts 2 points of input for 992 extra model calls.

The document line does earn its place. It added 6 chart answers on the Q1 supplement at full context and costs nothing.

Follow-up: the caption sentence, run F

F is E's 10 chunks with two prompt changes: the document identification laid out as a block (Title, Issuer, Form, Period, Year) instead of one line, and one sentence added after the exactness rule in the system prompt: Captions need not match the question's wording; the line must be the same item.

F against E	Same figure	Different figure	E answered, F declined	F answered, E declined	Net answers
F	254	1	14	8	−6

What the sentence did It recovered one caption case. "New originations, funded" answered from the Loan fundings row, $197,895, which E had declined for not matching the caption. Five of the other seven gains are the 10-Q chart percentages that E declined and every other run answered.

What it did not do The synonym misses it was written for are not caption misses. "First mortgages" was declined on both releases because the question cites Note 3 and a release has no notes. "Credit spread, senior and mezzanine" was declined on both supplements because no mezzanine line exists, though the portfolio is entirely senior loans. Neither is about the caption of the line, so the sentence had no purchase. F declined 14 answers E gave, of which 6 are E being right and F over-strict, 5 are E being wrong or derived and F correctly declining, and 3 are arguable.

Wrong figure Two of F's gains are wrong. North Carolina at 6% and Massachusetts at 12% on the Q4 supplement, read from a map whose labels do not name either state. E had declined both, naming the ambiguity. One further difference, first mortgages at $4,898,951 against $5,118,941, is the held-for-investment line against the total and is defensible either way.

Conclusion. The document block performs the same as the one-line form. The caption sentence recovered one answer, cost about six, and let two wrong map readings through. Withdraw it. The strictness it was meant to relax is not caption matching; it is Opus honouring qualifiers in the question, Note 3, "and mezzanine", "in millions", and that behaviour is what keeps wrong figures out.

What remains open

The 6 label-strictness losses in C are the same behaviour that keeps Opus from mis-mapping. Loosening one loosens the other. It has not been tested whether a prompt change can recover "term loans" without accepting "real estate debt".
The 7 chart percentages E declined and C answered were answered by the original 10-chunk run as well. E's declines there are run-to-run variation, not a defect of the full context.
Correctness of the 12 answers D gained was checked against citations and, for one, against the 10-K. It was not checked against the PDFs.

Sources: scratchpad files rerun_*.pipeC.json, rerun_*.pipeD.json, rerun_*.pipeE.json and the difference dumps diff_C_vs_E.txt and diff_D_vs_E.txt from session bc2f817a. Contexts were built from the hosted RDS as of 8 September 2026.