docs(experiment): record real-agent storage composition results

Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
committed 2026-10-16 21:35:50 +00:00
1 parent 577ca5a0ae
commit d4b615e2b9
3 files changed
+93 -4

No files matched your search

+4 -1
View File
@@ -22,7 +22,10 @@ adapter comparisons.
The [architecture storage study](storage-study.md) adds exact native register-name
and address-space queries, sharing one bounded immutable capture per lease.
It separates coordinates, widths and native map misses from hardware absence,
SSA equality or alias claims.
SSA equality or alias claims. Both real SSH models complete the extended task;
Code Mode uses 307,888 versus 415,450 cumulative input tokens and 142.2 versus
158.1 seconds, but more local CPU. This single pair does not establish a stable
advantage; the full typecheck gate remains resource-limited.
The [function-boundary study](function-boundary-study.md) adds independent native
prototype entries and return descriptors, separated from raw input varnodes.
@@ -92,3 +92,8 @@ and exact varnode_xref names without function decompilation. Width-qualified
coordinates compose with prototype/value/passing-storage facts. Real x86-64
and AArch64 OpenSSH catalog checks do not establish big-endian, alias, runtime
or full provider parity coverage.
Both real models complete the extended native storage/prototype/call task.
Code Mode reduces cumulative/uncached input and wall time in that pair but uses
more local CPU and more inner fact reads. The shared metadata capability is
verified separately from this adapter observation; earlier mixed pairs remain.
+84 -3
View File
@@ -107,6 +107,87 @@ events and verified payload cleanup.
The [question](storage-boundary-question.txt) and [oracle](storage-boundary-oracle.md)
extend the earlier native prototype task with exact storage-name and space
metadata requirements. Both complete actual catalogs are available. Model
results will be recorded only after the real isolated Codex/MCP runs and full
answer/source checks. This capability checkpoint does not establish a Code
Mode advantage or broad complex-software superiority.
runtime is frozen at `577ca5a0ae84d61a00f576feed7306b6cf54949c`; both isolated
Codex 0.161.0 runs request gpt-6.1-sol/high, in Direct → Code order. Neither
client feature flag establishes the available client tools: both lanes actually
have the common client JavaScript executor. This compares the MCP calling
adapters inside that common client surface.
Both complete answers pass the frozen oracle. They identify the owner/caller,
report all fourteen native prototype entries, native model/lock/return/error
flags, the 3/44 raw inputs and exact parameter matches, and preserve missing
matches without inferring original arity. Both use the native catalog for
CX versus RCX, R9D and the return RAX, distinguish R15 value storage from RDI
passing storage, and leave stack tuples without fabricated register names.
Code/register/stack descriptors remain separate metadata observations. Both
retain snapshot-local SSA identities, machine corroboration, unknowns and close.
Scripted checks verify every cited source against the delivered trace, retained
digest, target/engine identity, complete previous function records, prototypes
and exact input matches: ten Direct sources and twelve Code sources. Each lane
cites one native catalog and one capture per selected function. Additional
input-space counts, selected call argument/value/passing tuples and nine
independent ELF instruction sites agree; relative call/LEA destinations are
decoded from file-backed ELF segments. This source/fidelity check is separate
from reviewing the complete natural-language answer against the oracle; both
were performed. It does not prove all native recovery is machine-equivalent.
| Whole completed task | Direct | Code Mode |
| -------------------------------------------- | --------- | --------- |
| Required answer | pass | pass |
| Cumulative input tokens | 415,450 | 307,888 |
| Cached input, included above | 359,936 | 273,664 |
| Uncached input | 55,514 | 34,224 |
| Output tokens | 7,170 | 6,575 |
| Reasoning tokens, included in output | 1,802 | 1,226 |
| Actual unique model responses | 13 | 12 |
| Client executor / wait calls | 12 / 0 | 11 / 0 |
| MCP attempts, including discovery/open/close | 28 | 12 |
| Direct fact reads / Code inner API reads | 25 | 32 |
| Spaces / registers reads | 1 / 2 | 1 / 2 |
| Parameters reads | 2 | 4 |
| API / transport failures | 0 / 0 | 0 / 0 |
| Model wall seconds | 158.116 | 142.153 |
| Family wall seconds | 158.992 | 142.658 |
| Local family CPU seconds | 18.232034 | 20.391465 |
| Kernel family peak MiB | 672.0 | 636.9 |
All attempts are retained and charged; there are no failed model attempts or
retries in this pair. Summed unique per-response usage matches the runner totals.
Both use fresh leases/private native caches, preserve target/engine/runtime
fingerprints, remove isolated profiles and verify payload cleanup with zero OOM
events. Native catalog reuse is warm within the lease; remote model/OS caches,
resolved model build, remote compute and host contention remain uncontrolled
or unmeasured. The native caller capture itself took 4.432 s Direct / 5.859 s
Code; whole CPU differences cannot be attributed solely to adapter overhead.
In this one pair, Code uses 25.9% less cumulative input, 38.4% less uncached
input and 10.1% less model wall time, while local family CPU is 11.8% higher.
It performs more fact reads, including repeated cached prototypes and joins,
and only one fewer model response despite sixteen fewer MCP requests. The
earlier studies retain their mixed results. This new task adds metadata
requirements, so old question timings are not an equivalent before/after test.
No stable Code Mode advantage, traditional-provider superiority or general
complex-software effectiveness follows from this sample.
Local raw records are `ssh-storage-{direct,codemode}-v1/`, sibling `.log`,
`.trace.jsonl`, `.providers.json`, `.tools.json` and `-evidence/` under
`.cache/atomic-codemode/`. Derived checks are `storage-model-costs.json`,
`storage-answer-checks.json` and `storage-answer-review.json`. Final answer
SHA-256 values are Direct
`d5b64cfec3236cae4357160148578b597e7fe8115b62e57bc77fee7eb027d022`, Code
`0d2937dc1054e3e8405a4bbd428a3f753d2ab07c6a839d7ee798bfe7f642f69f`.
These hashes identify the reviewed answers, not an automated correctness proof.
## Resource-limited repository gate
The separate full `npm run check:fast` did not finish. The guarded run stopped
at the 3 GiB early threshold after 44.290 s; a serial retry using the installed
Turbo 2.11.7 documented `--concurrency=1` option also stopped there after
26.823 s during typecheck. Last sampled memory.events have zero OOM/kill/max
events. Payload-exit/final counters are unavailable for these aborted runs;
neither is reported as a passing full gate. The guards remained enabled and
the model trials ran after these checks terminated. Changed-module lint and
format checks, all 87 experimental JS tests and the real-provider checks above
pass. Production-wide type checking remains unverified within this resource
budget; no unrelated main code was changed to bypass it.