Compare commits

..
16 Commits
Author SHA1 Message Date
gensokyo-dev 6381aa0b1d fix(ghidra): scope rejected-input diagnostics to their request (#7140)
Scope expected Ghidra input failures to the active request after the authenticated startup handshake. Preserve operation output, identity and log coordinates while retaining cumulative output with its session owner; lifecycle and unexpected failures keep full diagnostics.

Share remote input-code classification with domain error projection. Verify startup rejection, healthy continuation, cleanup, unknown errors, OOM diagnostics and real CLI/MCP workflows. The measured unknown-symbol reply falls from 13,170 to 2,209 bytes with unchanged successful facts and Evidence IDs.
2026-10-25 22:37:32 +08:00
gensokyo-dev f69c52a1cf feat(native): page retained load-image observations (#7134)
Add optional native load-image summary delivery and retained selected-row pages through shared CLI/MCP analysis views. Preserve source verification, provenance, unavailable collections, exact row identity and opt-in byte/check payloads.

Preserve direct caller paths during shared CLI summary publication and cover raw/DOL projection contracts through the actual SDK. Real Ghidra ELF CLI/MCP captures, retained and file-backed pages, focused regressions and static checks passed; the PR records existing main schema failures and the JavaScript resource-recovery CI uncertainty.
2026-10-25 21:50:42 +08:00
Kaoru0822andClaude Opus 5.5 25f0f9934d fix(javascript): close member-path cycles through self stores (#7131)
A write below a slot that stores a value read from the same slot, such
as `s.c = s.c.p; s.c.i = 1` on a parameter, revisits the binding with
one more segment after the stored key on every turn. The cycle closure
only recognized segments added at the front of the path (`node =
node[key]`), so the work-list never terminated and exhausted the owned
worker heap (CodeMirror 5 `mode/stylus/stylus.js`).

Recognize a contiguous segment inserted after a stable base, check that
repeating it after that base can never be pruned, and escape through it
once, as the self-alias rule already does. Front insertions keep their
existing order and result.

Fixes #7130

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:32:29 -07:00
Kaoru0822andClaude Opus 5.5 70b96b65d9 fix(javascript): stop recording replaced slot values as store receivers (#7128)
Reference-store identity queries resolved the written target as a value,
so once a step consumed the written key the walk continued into the value
read from that slot and recorded it, and everything reachable from it, as
an object the store mutates. Every value stored into one local object at
unknown keys then received an entry for every such store, and each new
store re-queued every earlier one. Ambient-write indexing grew cubically
in the number of such stores.

Receiver queries now drop items whose projection no longer ends at the
written key. Stored-value selection has one owner, which the resolver and
the store fixpoint share. The fixpoint re-queues a dependent only when a
changed entry can reach it with a selection that its receiver query keeps.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:29:23 -07:00
Kaoru0822andClaude Opus 5.5 5a22ae6bc8 fix(mcp)!: return a compact binary_session status by default (#7127)
* fix(mcp)!: return a compact binary_session status by default

binary_session {} with no target open returned about 66 KB, most of it
provider rejection diagnostics and candidate capability sets that the
skill's recovery flow re-reads on every retry. 6.8.0 grew it by 21%
because the shared regex capability limitation began repeating setting
bounds and step-by-step recovery for every search operation of every
provider (#6099, #6267, #6301, #7049) and IDA gained that limitation for
its two search operations (#4862).

The regex limitation now states the effective deadline and heap, their
sources and where to change them; the deadline and heap errors keep the
full recovery. binary_session gains detail: summary (default) or
complete. The summary keeps open state, target identity, the binding,
route capabilities, run, activity and health status, each candidate's
availability and target-support status, code and reason, and every
tool's availability, reason, remediation and modes, and lists the
fields it leaves out in omitted_fields. Complete detail returns the
previous shape. The CLI capabilities and providers commands keep
complete provider facts through the same application projection.

BREAKING CHANGE: binary_session {} no longer returns
analysis_provider_candidates[].availability.diagnostics,
target_support.diagnostics or capabilities, or tool_availability group
surface, effects and annotations. Pass {"detail": "complete"} to get
them.

Fixes #7109

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): key binary_session output fields by detail

The output schema accepted detail: "complete" with summary candidates or
tool groups, and a summary without omitted_fields, so detail did not
tell callers which fields existed. Each session kind is now crossed
with a detail-specific shape: a summary requires omitted_fields and
summary-only candidates and tool groups; complete requires diagnostics,
candidate capabilities and static tool flags and forbids omitted_fields.
Both the canonical schema and the SDK-validated advertised schema reject
mismatched shapes.

The verifier helper accepts summary candidates only for a validated MCP
summary; CLI provider output still requires candidate capabilities.
Regex limitation tests check values, sources and setting names instead
of whole sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:29:03 -07:00
Kaoru0822andClaude Opus 5.5 e06be06530 fix(cli): report JSON Lines selection refusals as caller errors (#7124)
* fix(cli): report JSON Lines selection refusals as caller errors

A --format jsonl refusal because the selected value is not a record
collection is the caller's output selection, not unreadable analysis
output. Report it as invalid_request/invalid_input that names the
selection and its observed shape, keep the --format json /
--filter-output remediation, and drop the rea doctor advice.

AnalysisInputError gains an optional producer remediation, mirroring
the other refusal classes. The JSON Lines owner examines its documented
boundaries once for both admission and refusal facts.

docs/cli.md now states that columnar results such as
inspect-sqlite-database --table rows are not record collections.

Fixes #7110

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cli): carry JSON Lines refusal facts as structured observations

JSON Lines refusals now put the location and shape of each refused value
in details.issues[].observed, and the message is built from those facts.
Tests assert the classification, issue path, observed facts, and the
shared remediation constant instead of fixed sentences.

The filtered analysis-view export check now uses a non-empty request
and compares the emitted rows with the unfiltered export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:26:13 -07:00
Kaoru0822andClaude Opus 5.5 81afb043f7 fix(javascript): evaluate each short-circuit left operand once (#7126)
Logical branch selection evaluated the left operand to decide which
branches stay reachable, and value evaluation then evaluated the selected
left branch again. In a left-nested `a || b || c ...` chain every level
repeated that, so semantic analysis doubled per operand and a 24-operand
chain exhausted the per-source deadline.

The selection now carries the left operand's value with the branch, and
value evaluation reuses it. Provenance still walks the branch nodes.
Results are unchanged.

Fixes #7115

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:25:54 -07:00
Kaoru0822andClaude Opus 5.5 7336a566ac fix(mcp): fit every advertised input schema within the 64 KiB client limit (#7119)
* fix(mcp): fit every advertised input schema within the 64 KiB client limit

Claude Desktop drops a local connector tool whose compact-JSON input
schema exceeds 64 KiB. The full input profile advertised
compare_web_captures at 171,867 bytes and inspect_analysis_view at
71,632 bytes, because the self-contained projection inlines the
complete producer capture schemas once per side.

The full advertisement now fits the limit at its producing boundary.
A property whose schema repeats an earlier sibling's points at it, and
schemas below the deepest instance level that fits are summarized as
their types, size constraints and descriptions. Every rewrite widens the
advertisement through the existing relaxation rules for oneOf, not,
if/then/else, contains and unevaluated keywords, so canonical inputs
stay accepted and the canonical contract still validates every call.
Schemas within the limit are unchanged, and the compact profile is
still presented from the complete projection.

Numeric const and enum literals are also relaxed in the advertisement,
like boolean literals, since Gemini accepts only string enum members.

Fixes #4081

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mcp): keep fitted input summaries widening and at the deepest fitting level

Array summaries took item types only from `items`, so a tuple such as
`{prefixItems: [{type: string}], items: {type: number}}` was advertised
as a number array that rejected `["a", 1]`. Item types now include tuple
positions from `prefixItems`, a draft-07 `items` array, and
`additionalItems`, and stay unconstrained when later elements are open.

A summary can also be longer than the schema it replaces (a nullable
array becomes an anyOf), so candidate size is not monotone in the
retained depth and stopping at the first overflow could skip a deeper
candidate that fits. A schema is now kept whole when its summary is not
smaller, and fitting searches from the deepest level down.

Property tests check that every summary accepts what its schema accepts
over every catalog input subschema and synthetic tuples, and that the
fitted schema is the deepest fitting candidate. The browser comparison
boundary test now expects a malformed `after` capture to be rejected
canonically, since `after` is advertised as the same schema as `before`;
the same malformation on `before` stays advertised-invalid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:25:07 -07:00
Kaoru0822andClaude Opus 5.5 aa79f59478 fix(javascript): bind defaulted and destructured Webpack factory roles (#7117)
* fix(javascript): bind defaulted and destructured Webpack factory roles

A Webpack factory whose module parameter was defaulted (`e = {}`) or
destructured (`{exports: x}`) reported an exact empty export list with
complete coverage, because roles were taken only from plain identifiers.

Bind factory roles once, by position, in javascriptBundlerFactoryRoles.
Defaulted module/exports parameters keep their role (the convention always
supplies the argument), `{exports: alias}` binds the alias to
module.exports, and a repeated sloppy parameter name binds its last
argument. Other role patterns (rest, array, nested or computed keys, a
non-identifier loader) receive no role and count as an unknown finding,
so the file's analysis is partial instead of claiming no exports. The
export collector, the factory require name and the standalone bootstrap
externals all use these parsed roles.

Fixes #7113

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): keep factory receivers through unordered reads

A defaulted role parameter kept the unused default in the binding facts
passed to write ordering, so a receiver read inside a loop, nested block
or closure was unresolved and its export silently dropped while the
module stayed complete.

The role owner now removes defaults the calling convention never runs
before ordering. Separately, an export-shaped write through a role
binding whose read stays unordered now counts as an unknown finding
instead of being dropped, so an incomplete list is never reported as
complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): treat never-run factory parameter defaults as dead code

Removing unused defaults from binding facts let writes inside a default
expression reach the export collector: `(e = {}, t = (() => {
e.exports.ghost = 1; })())` reported `ghost` although Webpack supplies
both arguments. Loader calls in such defaults also produced require
references and async chunk keys.

The role owner now records the default of every role parameter, any
binding pattern included, as never run. The export collector, the
async-chunk walk and the static source traversal skip those subtrees,
so they yield no exports, references or chunk keys. Defaults of later,
non-role parameters still run and are still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:23:23 -07:00
Kaoru0822andClaude Opus 5.5 d32d9f2902 fix(lldb): keep the bridge's no-target proof through interruption (#7111)
When cancellation or the deadline interrupted LLDB after it had published
its bridge result, the tracer never read that result, so an explicit
`target_process_created: false` (or `target_process_exit_observed: true`)
was lost. The run kept a `native-target:unknown` marker that no cleanup
retry could release, wedging the tracer for every later trace.

Read the published result once LLDB's process group is stopped, even for
an interrupted run, and apply the same target-gone proof in cleanup retry.
Cancellation or timeout stays the primary outcome; a created target, a
truncated result, or a missing PID alone stays unresolved.

Fixes #5385

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:21:26 -07:00
Kaoru0822andClaude Opus 5.5 cc2656be8d perf(native): charge function matching for the copies it makes (#7108)
* perf(native): charge matching for what it copies, not borrowed retained inventories

Native function matching deep-walked every retained list_procedures record
at about 2.8 KB per Ghidra-shaped row, although the session ledger lends
those immutable records without a copy, and charged each inventory's one
Evidence ID once per function. The report and Evidence copies of every
emitted member were not charged at all.

Charge each inventory row before parsing for the member it becomes,
including its report and Evidence copies, charge the shared inventory ID
once, and skip the walk only for borrowed immutable inventories and closure
records that matching never parses. Inline Evidence, bundle records and
selected observations keep their existing charges. Retained references now
fit about 22,000 rows per side instead of 10,000, and an oversized side is
refused before its rows are parsed.

Refs #6406

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native): charge the envelope of borrowed matching inputs

Borrowed inventories skipped admission entirely, but matching copies their
limitations into the report and Evidence and digests their profile, so a
retained inventory with few rows and large limitations escaped the
allowance. Charge every envelope field and leave only the result payloads,
which row charging covers or matching never parses, uncharged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:21:09 -07:00
Kaoru0822andClaude Opus 5.5 58a5275aa2 fix(mcp): advertise reduced array elements with Gemini-compatible item types (#7107)
Reduced compact input schemas gave every array property an any-value item
schema, a six-member type list beside nested `items`. Gemini function
declarations accept `items` only beside the single `array` type, so every
request through a Gemini-backed client failed with
`properties[functions].items.items: field predicate failed: $type == Type.ARRAY`.

Reduced array properties now advertise the element types their canonical item
schemas accept. Unknown, tuple, and mixed array elements use the empty item
schema, and properties that may be arrays or another type list typed anyOf
branches. An inline `not: {}` keeps the empty schema. The compact catalog test
now rejects `items` beside a non-array type as well as untyped item unions.

Fixes #5201

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:20:36 -07:00
Kaoru0822andClaude Opus 5.5 0eb3919d69 fix(artifacts): admit raw UDRW disk images (#7106)
* fix(artifacts): admit raw UDRW disk images

Raw (UDRW, UDTO) disk images have no UDIF koly trailer, so target
admission and root inventory both refused them as unsupported binaries.
Recognize disk images in one domain owner (koly trailer, GPT, Apple
partition map, MBR, or an unpartitioned HFS+/HFSX volume header) that
both boundaries consume.

hdiutil reports "has no checksum" for these images; the native reader
now treats that status line as an explicit absent checksum, records the
nonzero verify command, attaches read-only, and reports that integrity
was not verified. Checksum mismatches and unrecognized images are still
refused.

Positional range reads used by these probes now fill until end of file
so short host reads cannot hide a signature.

Fixes #5687

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(artifacts): read disk image signature windows at the recognizer

Target admission passed its first unfilled 4 KiB read to the raw-image
checks, so a short host read could hide a GPT, APM, MBR or HFS+
signature that inventory (which fills its prefix) still observed. The
recognizer now reads its own leading window through the caller's
filling positional read, and target admission fills its format-probe
prefix as well. The controlled short-read regression now covers target
admission alongside inventory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:20:24 -07:00
Kaoru0822andClaude Opus 5.5 a8e5eaf1aa fix(macho): derive the runtime name of each Mach-O symbol (#7105)
inspect_macho reported only the symbol-table spelling (`_main`), while
LLDB, dlsym and observe_native_calls use the runtime spelling (`main`).
Passing a static name as a breakpoint silently selected a different
function, so results from the two tools could not be joined by name.

Keep `name` as the exact symbol-table fact and add a derived
`runtime_name` to every export and import row, produced by one shared
Mach-O symbol-name owner. Managed P/Invoke verification now uses the
same owner instead of its own underscore stripping.

Fixes #5298

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 05:19:54 -07:00
fc43e285ea feat(browser): retain compact scenario summaries and selected views (#7120)
Retain complete browser scenario Evidence while delivering compact summaries and selected step, event, transaction and text views through shared CLI/MCP workflows. Preserve initial CLI failures across cleanup retries and validate with real Chromium/TodoMVC.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 19:03:06 +08:00
morluto bc143125ed fix(release): reject missing manifest release boundaries 2026-10-25 17:25:18 +08:00
146 changed files with 9798 additions and 1012 deletions

No files matched your search

@@ -374,12 +374,16 @@ native target whose analysis tool operates on the active target. Do not call an
unadvertised tool; inspect `binary_session` with `{}` and its
`result.tool_availability` for availability reasons and remediation. Expand each
group's `names` into entries, inheriting the inventory-level client requirements
unless the group overrides them. In its closed effect and annotation vocabularies,
listed flags are true and omitted known flags are false. The grouped name union
is the complete tool inventory; use `tools/list` for canonical tool schemas.
Provider candidate facts describe possible providers, while top-level session
capabilities describe the current route and `analysis_provider_binding` names the
selected provider. Do not treat those scopes as interchangeable.
unless the group overrides them. The grouped name union is the complete tool
inventory; use `tools/list` for canonical tool schemas. The default summary keeps
each provider candidate's availability and target-support status, code and
reason; `omitted_fields` lists what it leaves out. Request `{"detail": "complete"}`
only when you need provider diagnostics, candidate capability sets, or tool
surface, effect and annotation flags; in those closed flag vocabularies, listed
flags are true and omitted known flags are false. Provider candidate facts
describe possible providers, while top-level session capabilities describe the
current route and `analysis_provider_binding` names the selected provider. Do
not treat those scopes as interchangeable.
For a Ghidra startup timeout on a large native target, follow
[deadline recovery](references/native-and-artifacts.md#ghidra-startup-deadline-recovery)
@@ -22,7 +22,7 @@ does not establish which server an approval wildcard covers. See
| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. |
| MCP connection dropped or REA could not start | If the connection drops, retry that call once only when it is safe to repeat or known not to have started. If it may already have changed the target, or the same failure returns, tell the user REA's connection failed and ask them to restart the MCP client. If REA could not start its connection, retry once, then tell the user and ask them to restart the MCP client. Do not retry a startup failure that returns the same unchanged result. Run `rea doctor` when REA could not start or tools stay missing after restart. Doctor cannot restore a live connection by itself. |
| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Expand each group's tool names and inherit the shared client requirements unless overridden. Use the operation's availability, reason, and remediation; a complete tool catalog does not imply every operation is available. |
| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Expand each group's tool names and inherit the shared client requirements unless overridden. Use the operation's availability, reason, and remediation; a complete tool catalog does not imply every operation is available. Request `{"detail": "complete"}` only for provider diagnostics. |
| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. |
If the client cannot launch its shell/sandbox helper or read the installed skill,
@@ -316,24 +316,28 @@ run, use `observe_native_calls` with explicit breakpoints and a bounded
`duration_ms`/`max_events`. It launches a new process under LLDB, so the target
runs and may change files or use the network. Hardened-runtime targets need the
`get-task-allow` entitlement.
Function breakpoints use exact LLDB runtime names: a Darwin C `_main` static symbol
may be `main` at runtime. REA does not translate names. Set `module` to the image
Function breakpoints use exact LLDB runtime names: pass an `inspect_macho` export
or import `runtime_name` (`main`), not its symbol-table `name` (`_main`). REA does
not translate breakpoint names. Set `module` to the image
file name when selecting that image only; otherwise loaded system images also
match. Inspect retained locations and module paths before attributing a call.
Use `binary_session` with no arguments to check the open target, selected
provider, and alignment. Its default `result.tool_availability` carries a
complete inventory grouped by shared facts. Expand each group's `names` and
inherit the inventory-level client requirements unless a group overrides them;
availability, reasons, remediation, modes, and flags remain inline. The
`effects` and `annotations` arrays name true flags from fixed vocabularies, so an
omitted known flag is false. `tools/list` remains the canonical source for full
tool schemas when target or provider state changes.
provider, and alignment. Its default summary `result.tool_availability` carries
the complete tool inventory grouped by current status. Expand each group's
`names` and inherit the inventory-level client requirements unless a group
overrides them; availability, reasons, remediation, and modes remain inline.
Provider candidates keep availability and target-support status, code, and
reason. `omitted_fields` names what the summary leaves out; `{"detail":
"complete"}` adds provider diagnostics, candidate capability sets, and tool
surface, `effects`, and `annotations`. Those arrays name true flags from fixed
vocabularies, so an omitted known flag is false. `tools/list` remains the
canonical source for full tool schemas when target or provider state changes.
Provider capability groups carry provider identity and ordered limitations.
Prepend each set's `shared_limitations` to a group's `limitations` to recover the
operation's full ordered list. Candidate capabilities describe possible
providers. Top-level session capabilities describe the current route, and
operation's full ordered list. Candidate capabilities, returned with complete
detail, describe possible providers. Top-level session capabilities describe the current route, and
`analysis_provider_binding` identifies the selected provider.
## Ghidra startup deadline recovery
@@ -426,6 +430,14 @@ loaded bytes, file mappings, relocations and the entry against the immutable
snapshot. Keep `mismatch` and `unsupported` explicit. Verified import covers the
reported static image, not runtime DOS or PC-98 hardware behavior.
For a large load-image report, select `detail: "summary"` and query its
`parent_evidence_id` with `inspect_analysis_view` kind `native-load-image`.
Choose a collection such as `relocations` or `mappings`, select only needed
fields, and page with offset/limit. Original indices preserve same-address
alternatives. Null collections remain unrecorded or inapplicable; paging does
not upgrade unsupported verification. CLI requires `--evidence_output` for
summary delivery so complete source Evidence remains accessible.
For Ghidra decompilation, `procedure_pseudo_code` also accepts `procedure: "file-offset:0x..."`; CLI `rea decompile <path> --file-offset=0x... --provider ghidra` resolves the source byte offset internally. An unresolved 0x-prefixed numeric procedure selector also falls back to a unique file mapping. Existing symbols and analysis addresses win; multiple mappings fail as ambiguous. Preserve the returned file-offset and address locations and raw selection observations. ELF virtual addresses are distinct from file offsets; never assume an arbitrary offset plus image base is an analysis address.
Complete function body ranges use inclusive ends and can contain gaps. Do not
@@ -86,6 +86,8 @@ Page-declared WebMCP tools are untrusted inventory and are never invoked by REA.
browser and profile. Check `browser.cleanup` for the outcome.
Request only the DOM, accessibility, or network details needed for the question.
The default final capture contains the URL; step outcomes are reported separately.
For multi-step flows with trees or events, select `detail: "summary"` and page
exact steps, transactions, text slices or events with `inspect_analysis_view` kind `browser-scenario`.
Add a wait action for the expected page state before capturing it. Report that
REA performed the declared actions.
+55
View File
@@ -621,3 +621,58 @@ rules and preserve the same missing fact.
Network normalization also commits the removal of capture-local transaction IDs,
attribution to the requesting step, and unordered transaction comparison. These
built-in rule identities participate in the normalization digest.
## Summary delivery and retained views
`capture_browser_scenario` accepts `detail: "summary"` to retain the complete
capture in the MCP session and return its summary. The default remains
`complete`. Delivery detail does not change scenario parameters or identity.
The summary reports each step's status, error, URL and captured/missing/truncated
section names. It omits `not_requested` sections and repeated `before_url` values.
The accessibility tree comes from the first unsuccessful step, or the final
step on success; when that step captured no tree, its nearest earlier captured
tree is identified explicitly. Event and network counts cover retained records,
with capture omissions reported separately.
For the one-shot CLI, save the complete Evidence before selecting a summary:
```bash
rea capture-browser-scenario scenario.json --detail summary --evidence_output /absolute/path/scenario-evidence.json --json
```
Use the returned `normalized_result.parent_evidence_id` with
`inspect_analysis_view` source `{kind: "retained-evidence", evidence_id: "..."}`.
For a CLI bundle, use an `evidence-file` source with its absolute path and the
parent Evidence ID. Views read the recorded capture; they never replay actions.
- `{kind: "summary"}` returns the same overview.
- `{kind: "browser-scenario", collection: "steps", step_indices: [2], artifacts: ["accessibility"], offset: 0, limit: 1}` returns selected recorded steps and artifacts.
- `{kind: "browser-scenario", collection: "events", event_kinds: ["response"], transaction_ids: ["request-2"], offset: 0, limit: 20}` pages selected events with original item indices.
Omitted filters select all; empty arrays select none. Pagination counts matching
retained rows. Exhausting a page does not establish capture completeness or
current application state.
Transaction pages use `{kind: "browser-scenario", collection: "transactions", limit: 20}`.
Their stable identity is the parent Evidence ID plus `transaction_id`; each row
keeps original event indices. Metadata events are returned by default. Select
`event_kinds: ["network-content"]` for recorded bodies and header values, or
`event_kinds: []` for identity and indices alone. A `step_indices` filter selects
a whole transaction when any retained event belongs to a selected step; this
is receipt attribution, not proof that the action caused the request.
For a large retained tree, request `{kind: "browser-scenario", collection: "text",
step_index: 10, artifact: "accessibility", offset: 0, limit: 2048}`. `dom` is also
supported. Offsets and limits count UTF-16 code units. The full artifact digest
and byte size remain attached to each page. Missing, truncated and unrequested
text stay explicit; source truncation counts are preserved without treating an
exhausted page as a complete capture.
Network summary counts include every retained transaction identity, including
those whose request event was dropped. Missing request events, conflicting
metadata and observed request failures have separate counts. `transactions` is
`null` when network observation was unrequested or its legacy selection is
unknown and there are no retained transactions; an observed empty inventory is
`0`. Full capture omissions remain available and each page carries source
completeness, dropped-event counts and omitted-transaction accounting.
+8 -3
View File
@@ -73,9 +73,14 @@ arbitrary nested graph array.
A top-level array emits its elements in source order. An object must have one
clear `rows`, `items`, `records`, or `occurrences` array, directly or under
`page`. Evidence uses the same boundaries inside `normalized_result`.
An arbitrary report or a scalar selection is refused with an actionable error;
use `--format json` for a complete document. If several collections exist,
select one with `--filter-output`, for example `normalized_result.rows`.
Any other selection is refused before a record is written, including an
arbitrary report, a scalar, a nested array elsewhere, and a columnar object such
as the `inspect-sqlite-database --table` `normalized_result.rows`
(`{columns, values}`). The refusal is the caller error `invalid_request`
(`invalid_input`) and names the selection and its observed shape; the analysis
itself succeeded, so use `--format json` for the complete document. If several
collections exist, select one with `--filter-output`, for example
`normalized_result.rows` for an analysis view.
Object collections finish with a control record:
+48
View File
@@ -30,6 +30,17 @@ signal. Confirm resource causes from process output or host/container records;
operation-scoped JVM `OutOfMemoryError` observations retain their separate resource
classification. Automatic kernel or cgroup OOM attribution is not implemented.
After the authenticated startup handshake, rejected inputs (`invalid_request`,
`not_found`, `ambiguous`, or an invalid function name) use
`diagnostics.diagnostic_scope: "request"` when their active output checkpoint is
available. They preserve the active request's
`failure_output`, target/profile identity and runtime/log coordinates, while
omitting cumulative launcher stdout/stderr and launch-support output. Startup,
process, protocol and other remote failures retain full session diagnostics.
The Ghidra client still owns its bounded cumulative output; this projection does
not clear it or alter session readiness. Log paths refer to the owned ephemeral
runtime and are removed by normal cleanup.
Provider selection and profile identity follow
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
provider stays bound to the target; runtime failure never selects another
@@ -160,6 +171,43 @@ have no file backing. ELF mappings are observed
without independent format-specific load-image verification, as the result's
status and limitations state.
For large relocation inventories, select MCP `inspect_native_load_image` with
`detail: "summary"`. It returns verification state, observed identity and counts,
while retaining complete Evidence. Use `normalized_result.parent_evidence_id`
with `inspect_analysis_view` and a view such as:
```json
{
"kind": "native-load-image",
"collection": "relocations",
"offset": 0,
"limit": 32,
"fields": ["address", "status", "type"]
}
```
Collections are `source_files`, `mappings`, `relocations`, `entry_points`,
`entry_context`, `checks` and DOL `sections`. Optional
`indices` select original row indices before paging; empty indices select none.
Fields are collection-specific. Relocation defaults omit byte payloads; check defaults omit potentially large `expected`/`observed` inventories. Request those fields explicitly when needed. Other collections default to all fields. Metadata recovery remains counted and retained in complete parent Evidence.
Parent Evidence, collection and `source_index` identify a row, including same-address
relocation alternatives. Null counts or rows mean unrecorded or inapplicable;
they never establish an empty observation. Exhausting a page does not upgrade
unsupported verification or runtime coverage. Pages query immutable retained
facts without reopening the target or repeating Ghidra work.
Each page carries the source's structured `verification_status` independently
of its page coverage.
CLI summary delivery requires a complete Evidence bundle:
```sh
rea inspect-native-load-image <path> --provider ghidra --detail summary --evidence_output /new/load-image.json --json
```
Read selected pages later through `inspect-analysis-view` using the returned
parent ID and an `evidence-file` source pointing to that bundle. Complete
delivery remains the default in both adapters.
An unsuccessful numeric procedure lookup reports the current Ghidra image base
and gives the CLI commands above alongside the corresponding MCP operations.
Repeat any target-format, architecture, and entry-point options used to open the
+11 -2
View File
@@ -426,8 +426,17 @@ and links whose callee was renamed by a minifier are not recognized.
Bundler adapters own callback roles: Webpack supplies module, exports, then
loader; esbuild CommonJS supplies exports, then module. Export extraction uses
those binding identities for assignments and declarations, so renaming either
parameter does not change the recovered names. Global property-definition
intrinsics require lexical and visible-write proof. Factory export declarations
parameter does not change the recovered names. The convention always supplies
every role argument, so a role parameter's default never runs: a defaulted
module or exports parameter keeps its role, and the default expression yields
no exports, references or chunk keys. `{exports: alias}` in the module position
binds `alias` to `module.exports`.
Other role patterns, such as rest, array, nested or computed-key patterns or a
non-identifier loader, receive no role and count as unknown findings, so the
file's analysis is partial rather than an exact empty export list. An
export-shaped write through a role binding whose value at that read cannot be
ordered against a replacement is also an unknown finding. Global
property-definition intrinsics require lexical and visible-write proof. Factory export declarations
also require the loader's `d` member to have no prior or unordered visible
replacement, including writes through lexical aliases. Unrelated loader members
do not invalidate declarations. Ordering describes declarations within one factory
+52 -12
View File
@@ -583,11 +583,22 @@ capabilities. The default result contains the complete inventory in groups that
share status facts. `tool_availability.client_requirements` is the baseline for
each group; a group's optional `client_requirements` replaces that baseline for
the group's tools. Expand each group's `names` into one tool entry, carrying its
surface, availability, reason, remediation, default-mode status, modes, and
client requirements. Compare the union of group names with `tools/list` when
checking that a client received the full catalog. An operation absent from all
groups is missing from the complete inventory; it is not an implicit
unavailable entry.
availability, reason, remediation, default-mode status, modes, and client
requirements. Compare the union of group names with `tools/list` when checking
that a client received the full catalog. An operation absent from all groups is
missing from the complete inventory; it is not an implicit unavailable entry.
`detail` selects the projection and the result echoes it. The default
`summary` keeps open state, target identity, the binding, current-route
`capabilities`, run, activity and health status, and each provider candidate's
identity, selection, and availability and target-support `status`, `code`, and
`reason`. It lists the fields it leaves out in `omitted_fields`: provider
`diagnostics`, candidate `capabilities`, and each tool group's static `surface`,
`effects`, and `annotations`. Omitted fields are not empty or false.
`detail: "complete"` returns them inline, groups tools by those flags too, and
has no `omitted_fields`. The output schema keys each result's candidate,
tool-group and omission fields by its `detail`. CLI `capabilities` and
`providers` always return complete provider facts.
`get_call_graph` and `trace_call_path` report `forward` and `backward` modes.
Each requires `procedure_address` plus `procedure_callees` or `procedure_callers`,
@@ -600,8 +611,8 @@ finish without reading relationships, even when neither traversal mode is
available. Discovery does not block that direct invocation or replace its typed
partial/error outcomes.
The `effects` and `annotations` arrays list the true members of their fixed
vocabularies. A known member omitted from either array is false. This applies
With complete detail, the `effects` and `annotations` arrays list the true
members of their fixed vocabularies. A known member omitted from either array is false. This applies
only to these closed flag sets; it does not make an omitted tool, provider, mode,
or limitation false. Keep availability reasons, remediation, client-feature
requirements, and mode facts inline with the grouped tool names. The optional
@@ -632,7 +643,8 @@ identity, availability, reason, rejection code, effects, and ordered
limitations.
`binary_session.analysis_provider_candidates` is authoritative for deep-engine
discovery. Each candidate's capability sets describe that possible provider;
discovery. Each candidate's capability sets, returned with complete detail,
describe that possible provider;
they do not select it. The top-level `capabilities` describe the current route,
while `analysis_provider_binding` identifies the concrete selected route. These
fields are not interchangeable. Target-free discovery is sorted by provider ID,
@@ -1208,16 +1220,34 @@ after SDK conversion and check advertised validation and actual calls.
Individual model APIs can impose additional nesting limits; complete producer
captures can exceed ten structural levels.
Claude Desktop drops a local connector tool, with a generic "problem with that
connector" warning, when its input schema serialized as compact JSON exceeds
64 KiB ([#4081](https://github.com/morluto/rea/issues/4081)). Every advertised
input schema therefore fits 65,536 UTF-8 bytes, including its examples. A
self-contained schema that would exceed the limit is fitted in two steps, and
its root description says it is summarized. First, a property whose schema
repeats an earlier sibling's, such as `after` beside `before`, is advertised as
its types and description plus "Same schema as `before`." Second, schemas below
the deepest instance level that still fits are advertised as their types, size
constraints and descriptions, unless that summary would be longer than the
schema itself. Combinator and conditional branches count as the
level of the instance they describe. Both steps only widen the advertisement:
canonical inputs and examples stay accepted, required names and nested
constraints below the cut are not advertised, and the canonical contract still
validates every call. Schemas within the limit are advertised unchanged. The
compact profile starts from the complete projection, not from the fitted one.
`search_bytes` advertises its pattern and mask as descriptive strings. Its
shared canonical validator scans byte pairs, whitespace, wildcards and aligned
masks without a repeating regular expression or temporary byte arrays. The
server rejects malformed or unconstrained patterns before provider dispatch;
the advertised strings alone do not express those rules.
Boolean and empty-string literal constraints in input schemas are relaxed only
in this advertisement. Boolean `const` becomes `type: boolean`; empty-string
`const` or an enum containing an empty string is widened without dropping the
accepted empty value. Defaults and descriptions remain intact.
Boolean, numeric and empty-string literal constraints in input schemas are
relaxed only in this advertisement. Boolean `const` becomes `type: boolean`;
numeric `const` keeps its numeric type; empty-string `const`, an enum containing
an empty string, and an enum containing a number are widened without dropping
an accepted value. Defaults and descriptions remain intact.
An affected `not` is omitted, conditional predicates and their branches are
omitted together, pure boolean-literal `oneOf` branches collapse to a boolean,
and other affected `oneOf` branches are presented as `anyOf` to retain branch
@@ -1269,6 +1299,16 @@ This addresses the emitted shape reported through a Gemini-backed gateway in
[#5201](https://github.com/morluto/rea/issues/5201). Gateway conversion and
live provider acceptance have not been verified for this projection.
Gemini function declarations accept `items` only beside the single `array`
type. Reduced array properties therefore advertise the element types their
canonical item schemas accept, such as `{"type": "object"}` for a union of
object actions, instead of an any-value type list with nested `items`. An
element whose type is unknown, a tuple, or an array alongside other types uses
the empty item schema, and a property that may be an array or another type
lists typed `anyOf` branches. A boundary test rejects any compact
advertisement that places `items` beside another type or uses an untyped item
union.
The compact profile renders the same tool set, output schemas, annotations,
and canonical server-side validation. It changes only the advertised input
JSON Schema:
+14 -2
View File
@@ -118,8 +118,20 @@ extract its matching Evidence record to a JSON file for the existing CLI
`inspect-analysis-view` `evidence-file` source. Input bundles are unchanged.
Representation admission is bounded at 96 MiB before cloning selected observations
and expanding matching indexes and repeated per-function source links. Work yields to cancellation. No worker/provider
concurrency is introduced; this budget is not a bound on total process RSS.
and expanding matching indexes and repeated per-function source links. Each
inventory row is charged before parsing for the member it becomes, including the
report and Evidence copies that member is serialized into, so an oversized side
is refused before its rows are parsed. Inline Evidence and bundle records are
charged for the copies their authentication or loading creates. An exact
session reference to retained Evidence is borrowed without a copy, so only
what matching derives from it is charged; the inventory's own Evidence ID is
charged once, not per function. Selected observations are charged in full.
Measured with Ghidra-shaped rows, about 20-character names and no
observations, about 22,000 rows per side fit through retained
`list_procedures` references, and about 10,000 through inline Evidence or CLI
`source_bundle` inputs. Work yields to
cancellation. No worker/provider concurrency is introduced; this budget is not
a bound on total process RSS.
Summary delivery happens after complete matching Evidence is constructed and
retained. It reduces the initial reply, but does not avoid that construction or
raise the allowance. There is no runtime override. Repeated provenance remains
+20 -9
View File
@@ -14,6 +14,10 @@ device platform. A chained-import row ending in `+0x...` is ambiguous in
dyld's text output: it may be an addend or part of a literal symbol name. REA
keeps the full expression in `raw_name`, sets `name` and `addend` to null, and
marks `name_status: "ambiguous"`; it does not synthesize a symbol identity.
Each import and export `name` is the exact symbol-table spelling, such as
`_main`; `runtime_name` is the derived name LLDB and `dlsym` use, such as `main`,
and is null when `name` is null. See [native call observation](#native-call-observation) for
breakpoint names.
## Start with a function
@@ -916,15 +920,22 @@ missing LLDB. `duration_ms` controls the later target observation window.
rea observe-native-calls /Applications/Example.app '{"breakpoints":[{"kind":"objc-method","class_name":"NSURLSession","selector":"dataTaskWithRequest:completionHandler:"},{"kind":"function","name":"open","module":"libsystem_kernel.dylib"}],"duration_ms":5000,"backtrace_frames":4}' --json
```
Function names select LLDB runtime symbols. On Darwin, a C function reported as
`_main` by `inspect_macho` or `nm` can resolve as `main` in LLDB. REA preserves the
exact selected name; it does not remove underscores or translate static names.
Use `{"kind":"function","name":"main","module":"Example"}` to limit that
runtime symbol to the executable image named `Example`, then inspect the reported
locations and image paths. Omitting `module` deliberately matches every loaded
image, including system libraries. A match outside the executable is valid for
that request and does not by itself make coverage partial. This C naming example
does not establish a translation rule for Swift, C++ or other mangled symbols.
Function names select LLDB runtime symbols. On Darwin, the Mach-O symbol table
spells language-level symbols with one leading underscore, which LLDB and `dlsym`
omit: `_main` is `main`, C++ `__ZN3rea3addEii` is `_ZN3rea3addEii`, and Swift
`_$s…` is `$s…`. Names without that underscore, such as `-[Class selector]`, are
unchanged. `inspect_macho` keeps each export and import `name` exactly as the
symbol table spells it and derives its `runtime_name`; pass `runtime_name` as the
breakpoint `name`. `observe_native_calls` preserves the exact selected name; it
does not remove underscores or translate static names, so `_main` selects a
different runtime symbol. Use `{"kind":"function","name":"main","module":"Example"}`
to limit that runtime symbol to the executable image named `Example`, then inspect
the reported locations and image paths. Omitting `module` deliberately matches
every loaded image, including system libraries. A match outside the executable is
valid for that request and does not by itself make coverage partial. Event and
location `symbol` values are LLDB display names, which are demangled for C++ and
Swift; join them to `inspect_macho` through `breakpoint_index` and the request
`name`, or through `file_address` and the export `address`.
Each event is an observation: thread, image, symbol, load and file address,
the raw integer argument registers, optional caller frames and, for an
+7
View File
@@ -112,6 +112,13 @@ creation or retry mutations. An internal GraphQL error does not establish API
quota exhaustion or prove that query size caused the failure. Publication
continues through the pinned release-please action.
A nonzero manifest version must resolve to a GitHub release or tag before
proposal history is collected. If publication has not created that boundary
yet, preparation stops before opening or updating a PR; wait for publication
before retrying. Missing manifest versions and `0.0.0` still allow an initial
release. This prevents historical `Release-As` directives from being replayed
as unreleased changes.
When upgrading release-please, review the upstream manifest implementation,
refresh the versioned patch and run the release proposal and workflow boundary
tests. Version mismatch or a patch that neither applies cleanly nor matches an
@@ -0,0 +1,63 @@
# Browser scenario summary verification
The verifier runs the actual compiled CLI/MCP and caller-selected Chromium against
TodoMVC's JavaScript ES5 distribution served on loopback. Ten actions add five
todos. Every step captures URL, DOM, accessibility and a screenshot; the oracle
checks all completed steps, the five authored labels, retained events and
transaction identities. Summary runs also read step, text and transaction pages
and authenticate an exported complete parent.
Download the public distribution at the pinned revision into a disposable folder:
```bash
export REA_TODOMVC_APP="$(mktemp -d)"
python3 - "$REA_TODOMVC_APP" <<'PY'
import hashlib, json, sys, urllib.request
from pathlib import Path
root = Path(sys.argv[1])
revision = "ff43b02e59dfa604386bb382034b2cd07c2bcd8a"
files = ["index.html", "app.js", "base.css", "base.js", "controller.js",
"helpers.js", "index.css", "model.js", "store.js", "template.js", "view.js"]
manifest = []
for name in files:
url = f"https://raw.githubusercontent.com/tastejs/todomvc/{revision}/examples/javascript-es5/dist/{name}"
with urllib.request.urlopen(url, timeout=30) as reply:
data = reply.read()
(root / name).write_bytes(data)
manifest.append({"path": name, "url": url, "bytes": len(data),
"sha256": hashlib.sha256(data).hexdigest()})
(root / "provenance.json").write_text(json.dumps({"revision": revision, "files": manifest}, indent=2))
PY
```
Build each selected checkout with the repository's pinned Node/npm toolchain.
Use new output directories; exports never overwrite an earlier run:
```bash
node experiments/analysis-efficiency/browser-scenario-summary.mjs \
--repository /absolute/path/to/baseline --mode complete \
--application "$REA_TODOMVC_APP" --browser /absolute/path/to/chromium \
--output /absolute/path/to/new-baseline-report
node experiments/analysis-efficiency/browser-scenario-summary.mjs \
--mode summary --application "$REA_TODOMVC_APP" \
--browser /absolute/path/to/chromium \
--output /absolute/path/to/new-summary-report
```
`cold` is the first capture in a fresh MCP process; `warm` repeats the capture in
that process with a new owned browser/profile. Neither label establishes cold OS
file caches. Summary mode intentionally performs additional retained queries.
Compare equivalent answers and include those calls when reporting task cost;
smaller replies alone do not establish a faster task.
Add `--verify-boundaries` to a separate summary run for a real failed-step focus,
CLI summary publication, and MCP text readback from the CLI's complete bundle.
Those extra checks make its whole-run resource cost unsuitable for comparison
against a complete-only run.
Reports retain the actual catalog, arguments, MCP text and complete Evidence.
When estimating tokens, name the tokenizer and count discovery, arguments and
all replies needed for the chosen task. Raw serialized catalog estimates are
not actual model usage or a measurement of a client's rendered tool definitions.
Measure CPU and peak process-family memory externally, state the accounting
method, and include cleanup; the verifier's elapsed phase excludes final teardown.
@@ -0,0 +1,412 @@
import assert from "node:assert/strict";
import { createServer } from "node:http";
import { readFile, writeFile, mkdir, readdir } from "node:fs/promises";
import { resolve, join, extname } from "node:path";
import { pathToFileURL, fileURLToPath } from "node:url";
import { createRequire } from "node:module";
import { performance } from "node:perf_hooks";
import { execFile } from "node:child_process";
import { promisify, parseArgs } from "node:util";
// A developer verifier: the caller selects the pinned TodoMVC distribution,
// browser executable, output directory and baseline/final repository checkout.
const { values } = parseArgs({
options: {
repository: {
type: "string",
default: fileURLToPath(new URL("../../", import.meta.url)),
},
mode: { type: "string", default: "summary" },
application: { type: "string" },
browser: { type: "string" },
output: { type: "string" },
port: { type: "string", default: "0" },
"verify-boundaries": { type: "boolean", default: false },
},
});
assert.ok(
values.application && values.browser && values.output,
"Pass --application <TodoMVC javascript-es5 dist> --browser <executable> --output <new directory>",
);
const repo = resolve(values.repository);
const mode = values.mode;
const output = resolve(values.output);
const appRoot = resolve(values.application);
const executablePath = resolve(values.browser);
assert.ok(["complete", "summary"].includes(mode));
assert.ok(
!values["verify-boundaries"] || mode === "summary",
"--verify-boundaries requires summary mode",
);
process.chdir(repo);
const require = createRequire(join(repo, "package.json"));
const { Client } = await import(
pathToFileURL(require.resolve("@modelcontextprotocol/client")).href
);
const { StdioClientTransport } = await import(
pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio")).href
);
await mkdir(output);
const files = await readdir(appRoot);
const server = createServer(async (request, response) => {
try {
const name =
decodeURIComponent(
new URL(request.url, "http://127.0.0.1").pathname,
).slice(1) || "index.html";
assert.ok(files.includes(name));
const data = await readFile(join(appRoot, name));
response.setHeader(
"Content-Type",
{ ".html": "text/html", ".js": "text/javascript", ".css": "text/css" }[
extname(name)
] ?? "application/octet-stream",
);
response.end(data);
} catch {
response.writeHead(404);
response.end();
}
});
await new Promise((done, reject) => {
server.once("error", reject);
server.listen(Number(values.port), "127.0.0.1", done);
});
const address = server.address();
assert.ok(address && typeof address === "object");
const origin = `http://127.0.0.1:${address.port}`;
const environment = {
PATH: process.env.PATH,
NODE_OPTIONS: "--max-old-space-size=3072",
UV_THREADPOOL_SIZE: "2",
REA_LOG_LEVEL: "silent",
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
};
const transport = new StdioClientTransport({
command: process.execPath,
args: [join(repo, "scripts/rea.mjs"), "mcp"],
cwd: repo,
env: environment,
stderr: "pipe",
});
let stderr = "";
transport.stderr?.on("data", (chunk) => {
if (stderr.length < 65536) stderr += chunk;
});
const client = new Client({
name: "rea-browser-scenario-summary-verifier",
version: "1",
});
const calls = [];
const invoke = async (name, args, tools) => {
const toolDefinition = tools.find((tool) => tool.name === name);
assert.ok(
toolDefinition?.outputSchema,
`Missing named output schema: ${name}`,
);
const started = performance.now();
const result = await client.callTool(
{ name, arguments: args },
{ toolDefinition, timeout: 90000 },
);
calls.push({
name,
arguments: args,
content: result.content,
elapsed_ms: performance.now() - started,
});
assert.notEqual(result.isError, true, JSON.stringify(result));
return result.structuredContent;
};
const allStarted = performance.now();
try {
await client.connect(transport);
const discoveryStarted = performance.now();
const catalog = await client.listTools();
const discoveryMs = performance.now() - discoveryStarted;
await writeFile(join(output, "catalog.json"), JSON.stringify(catalog));
const actions = Array.from({ length: 5 }, (_, index) => [
{
step_id: `fill_${index}`,
action: "fill",
locator: { kind: "css", selector: ".new-todo" },
value: { source: "literal", value: `Review evidence ${index + 1}` },
timeout_ms: 10000,
},
{
step_id: `add_${index}`,
action: "press",
locator: { kind: "css", selector: ".new-todo" },
key: "Enter",
timeout_ms: 10000,
},
]).flat();
const scenario = {
browser: {
mode: "launch",
executable_path: executablePath,
headless: true,
},
start_url: { url: origin, query: [] },
actions,
capture: {
after_each_step: ["url", "dom", "accessibility", "screenshot"],
at_end: ["url", "dom", "accessibility"],
events: ["network"],
},
};
const runs = [];
for (const temperature of ["cold", "warm"]) {
const callStart = calls.length;
const result = await invoke(
"capture_browser_scenario",
{ ...scenario, ...(mode === "summary" ? { detail: "summary" } : {}) },
catalog.tools,
);
const parentId =
mode === "summary"
? result.normalized_result.parent_evidence_id
: result.evidence_id;
const bundlePath = join(output, `${temperature}-evidence.json`);
await invoke("export_evidence_bundle", { path: bundlePath }, catalog.tools);
const bundle = JSON.parse(await readFile(bundlePath, "utf8"));
const parent = bundle.records.find((row) => row.evidence_id === parentId);
assert.ok(parent);
const capture = parent.normalized_result;
assert.equal(capture.steps.length, 11);
assert.ok(capture.steps.every((step) => step.status === "completed"));
assert.equal(capture.browser.cleanup, "terminated-owned-process");
const final = capture.steps.at(-1).artifacts.accessibility;
assert.equal(final.state, "captured");
for (let index = 1; index <= 5; index++)
assert.ok(final.value.text.includes(`Review evidence ${index}`));
assert.ok(final.value.text.includes('strong: "5"'));
assert.ok(final.value.text.includes("text: items left"));
const transactionIds = new Set(
capture.events.items
.filter((event) => "transaction_id" in event)
.map((event) => event.transaction_id),
);
const details = {
temperature,
parentId,
steps: capture.steps.length,
retained_events: capture.events.retained,
transactions: transactionIds.size,
};
if (mode === "summary") {
const summary = result.normalized_result.summary;
assert.equal(summary.steps.length, 11);
assert.equal(
summary.final_accessibility.accessibility.value.text,
final.value.text,
);
assert.equal(summary.events.retained, capture.events.retained);
assert.equal(summary.limitation_count, parent.limitations.length);
assert.equal(summary.network.transactions, transactionIds.size);
const source = { kind: "retained-evidence", evidence_id: parentId };
const pagingStarted = performance.now();
for (const stepIndex of [1, 5, 10]) {
const page = await invoke(
"inspect_analysis_view",
{
source,
view: {
kind: "browser-scenario",
collection: "steps",
step_indices: [stepIndex],
artifacts: [],
limit: 1,
},
},
catalog.tools,
);
assert.equal(
page.normalized_result.items.rows[0].step_id,
capture.steps[stepIndex].step_id,
);
}
details.warm_page_total_ms = performance.now() - pagingStarted;
const text = await invoke(
"inspect_analysis_view",
{
source,
view: {
kind: "browser-scenario",
collection: "text",
step_index: 10,
artifact: "accessibility",
offset: 0,
limit: 32,
},
},
catalog.tools,
);
assert.equal(
text.normalized_result.items.text,
final.value.text.slice(0, 32),
);
const tx = await invoke(
"inspect_analysis_view",
{
source,
view: {
kind: "browser-scenario",
collection: "transactions",
limit: 2,
},
},
catalog.tools,
);
assert.deepEqual(
tx.normalized_result.items.rows.map((row) => row.transaction_id),
[...transactionIds].slice(0, 2),
);
details.returned_summary = summary;
}
details.calls = calls.slice(callStart);
runs.push(details);
}
const boundaryChecks = [];
if (values["verify-boundaries"]) {
const failed = await invoke(
"capture_browser_scenario",
{
...scenario,
actions: [
{
step_id: "missing",
action: "click",
locator: {
kind: "css",
selector: "#rea-control-that-does-not-exist",
},
timeout_ms: 100,
},
],
capture: {
after_each_step: ["url", "accessibility"],
at_end: ["url", "accessibility"],
events: ["network"],
},
detail: "summary",
},
catalog.tools,
);
assert.equal(failed.normalized_result.summary.steps[1].status, "failed");
assert.equal(
failed.normalized_result.summary.final_accessibility.focus_step_index,
1,
);
assert.equal(
failed.normalized_result.summary.completeness.equality_eligible,
false,
);
boundaryChecks.push("real failed-step focus and incomplete coverage");
const execute = promisify(execFile);
const bundlePath = join(output, "cli-evidence.json");
const { stdout } = await execute(
process.execPath,
[
join(repo, "scripts/rea.mjs"),
"capture-browser-scenario",
JSON.stringify(scenario),
"--detail",
"summary",
"--evidence_output",
bundlePath,
"--json",
],
{
cwd: repo,
env: environment,
maxBuffer: 64 * 1024 * 1024,
timeout: 90000,
},
);
const cli = JSON.parse(stdout);
const parentId = cli.summary.normalized_result.parent_evidence_id;
const bundle = JSON.parse(await readFile(bundlePath, "utf8"));
const parent = bundle.records.find((row) => row.evidence_id === parentId);
assert.ok(parent);
assert.equal(parent.normalized_result.steps.length, 11);
assert.ok(
parent.normalized_result.steps.every(
(step) => step.status === "completed",
),
);
assert.ok(!("detail" in parent.parameters));
assert.equal(
cli.summary.normalized_result.summary.final_accessibility.accessibility
.value.text,
parent.normalized_result.steps.at(-1).artifacts.accessibility.value.text,
);
const text = await invoke(
"inspect_analysis_view",
{
source: {
kind: "evidence-file",
path: bundlePath,
evidence_id: parentId,
},
view: {
kind: "browser-scenario",
collection: "text",
step_index: 10,
artifact: "accessibility",
offset: 0,
limit: 32,
},
},
catalog.tools,
);
assert.equal(
text.normalized_result.items.text,
parent.normalized_result.steps
.at(-1)
.artifacts.accessibility.value.text.slice(0, 32),
);
boundaryChecks.push(
"real CLI summary, complete bundle and MCP evidence-file readback",
);
}
await writeFile(
join(output, "result.json"),
JSON.stringify(
{
repo,
mode,
application: appRoot,
browser: executablePath,
input_paths: {
repository: values.repository,
application: values.application,
browser: values.browser,
output: values.output,
},
origin,
discovery_ms: discoveryMs,
elapsed_ms: performance.now() - allStarted,
runs,
boundaryChecks,
stderr,
},
null,
2,
),
);
console.log(
JSON.stringify({
mode,
discovery_ms: discoveryMs,
runs: runs.map(
({ calls: _calls, returned_summary: _summary, ...rest }) => rest,
),
boundaryChecks,
elapsed_ms: performance.now() - allStarted,
}),
);
} finally {
await client.close();
server.closeAllConnections();
await new Promise((done) => server.close(done));
}
@@ -0,0 +1,89 @@
# Native load-image delivery verification
`native-load-image-views.mjs` runs the compiled production stdio MCP server with
a caller-selected Ghidra and JDK. It compares complete delivery with a summary
plus selected relocation pages, then exports and authenticates the complete
parent. Every call uses its actual advertised SDK contract. Both cold and warm
reads run in one fresh server; OS file caches are not declared cold.
Build each selected runtime with the pinned toolchain. Use absolute paths or
paths relative to the verifier's launch directory, and new output directories:
```sh
node experiments/analysis-efficiency/native-load-image-views.mjs \
--runtime /path/to/baseline --mode complete \
--target /usr/lib/x86_64-linux-gnu/libcrypto.so.3 \
--ghidra /path/to/ghidra_12.1.4_PUBLIC --java /path/to/jdk-21 \
--output /new/baseline-report
node experiments/analysis-efficiency/native-load-image-views.mjs \
--runtime /path/to/feature --mode summary \
--target /usr/lib/x86_64-linux-gnu/libcrypto.so.3 \
--ghidra /path/to/ghidra_12.1.4_PUBLIC --java /path/to/jdk-21 \
--output /new/summary-report --control /new/baseline-report
```
The question is the verification state, all collection counts, and the first
four relocation addresses/statuses. The summary path additionally checks the
last relocation's recorded bytes and unavailable independent verification
checks. It never treats ELF observations as verified. Complete delivery can
exceed the MCP response limit: the baseline then follows the reported retained
reference, exports its source and selects facts locally. That recovery is an
extra workflow, not a successful complete inline MCP reply. Include its calls
and local reads when comparing completed tasks.
The independent GNU `readelf -Wr` oracle checks that every file-table relocation
coordinate survives, including multiplicity. `readelf -Wl` supplies the original
PT_LOAD base independently of Ghidra's chosen import base. Producer order can
differ. Additional provider observations are retained and reported separately;
this oracle does not independently verify their origin. Full observation
digests and selected facts must agree with the control run.
The target SHA-256, native oracle output, raw catalog, arguments/replies,
complete Evidence and phase durations stay in the output directory. Measure
whole-process-family elapsed time, CPU and peak memory externally, with bounded
CPU/heap and a memory stop condition. The child requests a 1.5 GiB Ghidra heap
and a 3 GiB Node heap; neither is a whole-family RSS bound. Estimate tokens over
catalog discovery, arguments, all replies and local recovery facts using a named
tokenizer. Raw serialized schema estimates are not actual model usage; protocol
framing, prompts and client rendering remain unmeasured.
The shared receipt finalizer closes both transports, removes its owned workspace
after successful cleanup, and preserves cleanup/publication failures in the
report. Its phase elapsed field precedes final receipt publication; external
whole-family measurements must include that finalization.
## Recorded Linux x64 verification
One fresh cold/warm session per revision used Node 24.22.1, Ghidra 12.1.4,
two CPUs and the system `libcrypto.so.3` with SHA-256
`76dd3d93e5ee48950a92a58d59b94de8143847f91a80d9682c938767b991577d`.
Baseline `42382651b` has the same tree as main `fc43e285e`; feature `7a53a4a7f`.
Both retained the same parent ID and 21,118 observations. GNU readelf declared
21,117 relocation coordinates, all present; one provider-only coordinate at
relative `0x47dc60` remained visible without an independently established origin.
| Whole two-capture harness | Complete + local recovery | Summary + selected reads |
| --------------------------------------- | -----------------------------: | -----------------------: |
| Direct capture reply | Refused at the 10 MiB boundary | 9,787 bytes |
| Complete parent Evidence | 9,114,888 bytes | 9,114,888 bytes |
| Raw serialization, cl100k_base estimate | 912,927 tokens | 936,730 tokens |
| Elapsed | 164.93 s | 169.88 s |
| Process-family CPU | 281.75 s | 293.06 s |
| Sampled process-family RSS peak | 2.01 GiB | 2.30 GiB |
Token estimates include one raw catalog discovery, call arguments/replies and
baseline local recovery facts. Local command text, prompts, protocol framing
and client rendering were not measured; these are not model-usage totals.
The summary harness also checks tail bytes and unavailable checks through
additional RPCs. Neither lower task tokens, faster analysis nor lower memory
is established by this single pair. The benefit is direct bounded MCP access
to facts that previously required an export and local selection. RSS summed
the owned process family every 100 ms; shared pages can be double-counted and
short peaks missed. OS caches were warm.
The real CLI published the same complete parent with summary delivery, then a
separate CLI process read selected first/last rows from that bundle without
configured Ghidra/JDK. That file-page read took 4.83 s, 5.89 CPU seconds and
0.51 GiB sampled family RSS. This verifies historical projection independently
of reopening a native provider. Other hosts, engines and formats were not
verified in this experiment.
@@ -0,0 +1,357 @@
/** Actual Ghidra/MCP load observations with retained-page and readelf oracles. */
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import { createWriteStream } from "node:fs";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { finished } from "node:stream/promises";
import { parseArgs, promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
const { values } = parseArgs({
options: Object.fromEntries(
["runtime", "target", "output", "ghidra", "java", "mode", "control"].map(
(name) => [name, { type: "string" }],
),
),
});
for (const key of ["runtime", "target", "output", "ghidra", "java"]) {
assert.ok(values[key], `Missing --${key}`);
values[key] = resolve(values[key]);
}
if (values.control) values.control = resolve(values.control);
assert.ok(["complete", "summary"].includes(values.mode));
await mkdir(values.output);
const hash = (value) => createHash("sha256").update(value).digest("hex");
const { parseEvidence } = await import(
pathToFileURL(join(values.runtime, "dist/domain/evidence.js")).href
);
const { stdout: revision } = await promisify(execFile)("git", [
"-C",
values.runtime,
"rev-parse",
"HEAD",
]);
const { stdout: workingState } = await promisify(execFile)("git", [
"-C",
values.runtime,
"status",
"--porcelain",
]);
const targetSha256 = hash(await readFile(values.target));
const { stdout: oracle } = await promisify(execFile)(
"readelf",
["-Wr", values.target],
{ maxBuffer: 64 * 1024 ** 2 },
);
const relocationOffsets = [
...oracle.matchAll(/^([0-9a-f]+)\s+[0-9a-f]+\s+R_\S+/gm),
].map((row) => row[1]);
await writeFile(join(values.output, "readelf-relocations.txt"), oracle);
const { stdout: programHeaders } = await promisify(execFile)(
"readelf",
["-Wl", values.target],
{ maxBuffer: 1024 ** 2 },
);
const loadAddresses = [
...programHeaders.matchAll(/^\s*LOAD\s+0x[0-9a-f]+\s+0x([0-9a-f]+)/gm),
].map((row) => BigInt(`0x${row[1]}`));
assert.ok(loadAddresses.length > 0, "Expected an ELF load mapping");
const lowestLoad = loadAddresses.reduce((left, right) =>
left < right ? left : right,
);
await writeFile(
join(values.output, "readelf-program-headers.txt"),
programHeaders,
);
const isolatedHome = await mkdtemp(join(tmpdir(), "rea-load-image-verifier-"));
const transport = new StdioClientTransport({
command: process.execPath,
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
cwd: values.runtime,
env: {
HOME: isolatedHome,
PATH: process.env.PATH,
NODE_OPTIONS: "--max-old-space-size=3072",
GHIDRA_INSTALL_DIR: values.ghidra,
JAVA_HOME: values.java,
GHIDRA_HEADLESS_MAXMEM: "1536M",
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
REA_LOG_LEVEL: "silent",
},
stderr: "pipe",
});
const serverLog = createWriteStream(join(values.output, "server.log"));
const serverLogFinished = finished(serverLog);
serverLogFinished.catch(() => {});
transport.stderr?.pipe(serverLog);
const client = new Client({ name: "native-load-image-views", version: "1" });
const transcript = [];
const report = {
mode: values.mode,
runtime: values.runtime,
revision: revision.trim(),
dirty: workingState.length > 0,
artifact: { path: values.target, sha256: targetSha256 },
oracle: {
command: ["readelf", "-Wr", values.target],
sha256: hash(oracle),
relocation_count: relocationOffsets.length,
},
question:
"Report retained load-image verification status, collection counts, and the first four relocation addresses/statuses with source linkage.",
calls: [],
runs: [],
};
const call = createRetainedQueryCaller(client, (receipt, event) => {
report.calls.push(receipt);
transcript.push(event);
});
const result = async (name, input, phase = name) => {
const reply = await call(name, input, phase);
assert.notEqual(reply.isError, true, JSON.stringify(reply));
assert.ok(reply.structuredContent);
return reply.structuredContent;
};
const exportedParent = async (id, label) => {
const path = join(values.output, `${label}-evidence.json`);
await result("export_evidence_bundle", { path }, `${label}:export`);
const bundle = JSON.parse(await readFile(path, "utf8"));
const parent = bundle.records.find((record) => record.evidence_id === id);
assert.ok(parent, `Missing retained source ${id}`);
assert.equal(parent.operation, "inspect_native_load_image");
assert.equal(parent.subject.digest.sha256, targetSha256);
assert.equal(parent.provider.id, "ghidra");
return parseEvidence(parent);
};
const selectedRecords = (rows) =>
rows.slice(0, 4).map(({ address, status }) => ({ address, status }));
let opened = false;
const started = performance.now();
try {
await client.connect(transport);
const catalogStart = performance.now();
const catalog = await client.listTools();
assert.equal(catalog.nextCursor, undefined);
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
report.catalog = {
elapsed_ms: performance.now() - catalogStart,
bytes: Buffer.byteLength(JSON.stringify(catalog)),
sha256: hash(JSON.stringify(catalog)),
};
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
const original = client.callTool.bind(client);
client.callTool = (input, options) =>
original(input, {
...options,
timeout: 600000,
toolDefinition: definitions.get(input.name),
});
await result("open_binary", { path: values.target, provider_id: "ghidra" });
opened = true;
for (const temperature of ["cold", "warm"]) {
const runStart = performance.now();
const reply = await call(
"inspect_native_load_image",
values.mode === "summary" ? { detail: "summary" } : {},
temperature,
);
let parent;
let delivered;
let id;
if (reply.isError) {
assert.equal(values.mode, "complete");
const text = reply.content.find((item) => item.type === "text");
assert.ok(text);
const error = JSON.parse(text.text).error;
assert.equal(error.code, "resource_constraint");
assert.equal(error.details.reported_limits.boundary, "mcp-response");
id = error.details.reported_limits.evidence_reference.evidence_id;
parent = await exportedParent(id, temperature);
delivered = parent.normalized_result;
} else {
assert.ok(reply.structuredContent);
delivered = reply.structuredContent.normalized_result;
id =
values.mode === "summary"
? delivered.parent_evidence_id
: reply.structuredContent.evidence_id;
parent = await exportedParent(id, temperature);
if (values.mode === "complete")
assert.deepEqual(parent, reply.structuredContent);
}
const source = parent.normalized_result;
const observed = source.observations;
// Every file-table coordinate must survive, including repeated coordinates.
// Provider-added observations are retained separately; this oracle does not
// independently establish why the provider created them.
const base = BigInt(observed.image_base.split(":").at(-1));
const counts = new Map();
for (const row of observed.relocations) {
const key = (BigInt(row.address.split(":").at(-1)) - base).toString(16);
counts.set(key, (counts.get(key) ?? 0) + 1);
}
for (const offset of relocationOffsets) {
const key = (BigInt(`0x${offset}`) - lowestLoad).toString(16);
const remaining = counts.get(key) ?? 0;
assert.ok(remaining > 0, `Missing ELF relocation coordinate 0x${key}`);
counts.set(key, remaining - 1);
}
const providerOnly = [...counts]
.filter(([, count]) => count > 0)
.map(([relative_address, count]) => ({
relative_address: `0x${relative_address}`,
count,
}));
const facts = {
observation_sha256: hash(JSON.stringify(observed)),
verification: {
status: source.status,
...(source.status === "unsupported"
? { reason: source.reason }
: { format: source.format }),
},
counts: {
source_files: observed.source_files.length,
mappings: observed.mappings.length,
relocations: observed.relocations.length,
entry_points: observed.entry_points.length,
entry_context: observed.entry_context.length,
metadata_recovery: observed.metadata_recovery?.length ?? null,
checks: source.checks?.length ?? null,
sections: source.sections?.length ?? null,
},
records: selectedRecords(observed.relocations),
};
if (values.mode === "summary") {
assert.equal(delivered.parent_operation, parent.operation);
assert.equal(delivered.parent_digest, parent.evidence_id.slice(3));
assert.equal(delivered.summary.verification.status, source.status);
assert.deepEqual(delivered.summary.counts, facts.counts);
assert.ok(!Object.hasOwn(delivered.summary.observation, "relocations"));
const selected = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: {
kind: "native-load-image",
collection: "relocations",
offset: 0,
limit: 4,
fields: ["address", "status"],
},
},
`${temperature}:selected-relocations`,
);
assert.deepEqual(
selected.normalized_result.items.rows,
facts.records.map((record, source_index) => ({ source_index, record })),
);
assert.equal(
selected.normalized_result.verification_status,
source.status,
);
assert.equal(
selected.normalized_result.coverage.total,
observed.relocations.length,
);
const tail = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: {
kind: "native-load-image",
collection: "relocations",
indices: [observed.relocations.length - 1],
fields: ["address", "original_bytes_hex", "memory_bytes_hex"],
},
},
`${temperature}:tail-relocation`,
);
const last = observed.relocations.at(-1);
assert.deepEqual(tail.normalized_result.items.rows, [
{
source_index: observed.relocations.length - 1,
record: {
address: last.address,
original_bytes_hex: last.original_bytes_hex,
memory_bytes_hex: last.memory_bytes_hex,
},
},
]);
const checks = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: { kind: "native-load-image", collection: "checks", limit: 1 },
},
`${temperature}:verification-checks`,
);
assert.equal(source.status, "unsupported");
assert.equal(checks.normalized_result.items.rows, null);
}
assert.deepEqual(parent.parameters, {});
await writeFile(
join(values.output, `${temperature}-facts.json`),
JSON.stringify(facts),
);
if (values.control)
assert.deepEqual(
facts,
JSON.parse(
await readFile(
join(values.control, `${temperature}-facts.json`),
"utf8",
),
),
);
report.runs.push({
temperature,
parent_evidence_id: id,
source_provider: parent.provider,
source_analysis_profile_digest: parent.analysis_profile?.digest ?? null,
reply_refused: reply.isError === true,
complete_evidence_bytes: Buffer.byteLength(JSON.stringify(parent)),
facts_sha256: hash(JSON.stringify(facts)),
provider_only_relocation_coordinates: providerOnly,
elapsed_ms: performance.now() - runStart,
});
}
assert.equal(hash(await readFile(values.target)), targetSha256);
report.status = "passed";
} catch (cause) {
report.status = "failed";
report.failure = String(cause?.stack ?? cause);
process.exitCode = 1;
} finally {
if (opened) {
try {
await result("close_binary", {});
} catch (cause) {
report.status = "failed";
report.close_failure = String(cause);
process.exitCode = 1;
}
}
report.elapsed_before_finalization_ms = performance.now() - started;
if (
!(await finalizeRetainedQueryReceipt({
report,
client,
transport,
account: isolatedHome,
serverLog,
serverLogFinished,
output: values.output,
transcript,
}))
)
process.exitCode = 1;
}
+19 -4
View File
@@ -105,7 +105,22 @@
}
--- a/build/src/manifest.js
+++ b/build/src/manifest.js
@@ -261,9 +261,20 @@
@@ -248,6 +248,14 @@
releasesFound++;
}
}
+ // A recorded version requires its release boundary, even while a
+ // concurrent publisher has not created the matching release/tag yet.
+ for (const path of Object.keys(strategiesByPath)) {
+ const expectedVersion = this.releasedVersions[path];
+ if (expectedVersion && expectedVersion.toString() !== '0.0.0' && !releasesByPath[path]) {
+ throw new Error(`Missing expected release for path ${path} at version ${expectedVersion.toString()}: no matching release or tag found. Refusing to bootstrap an already released package.`);
+ }
+ }
const needsBootstrap = releasesFound < expectedReleases;
if (releasesFound < expectedReleases) {
this.logger.warn(`Expected ${expectedReleases} releases, only found ${releasesFound}`);
@@ -261,9 +269,20 @@
this.logger.info('Collecting commits since all latest releases');
const commits = [];
this.logger.debug(`commit search depth: ${this.commitSearchDepth}`);
@@ -127,7 +142,7 @@
batchSize: this.commitBatchSize,
});
const releaseShas = new Set(Object.values(releaseShasByPath));
@@ -272,6 +283,7 @@
@@ -272,6 +291,7 @@
// sha => release pull request
const releasePullRequestsBySha = {};
let releaseCommitsFound = 0;
@@ -135,7 +150,7 @@
for await (const commit of commitGenerator) {
if (releaseShas.has(commit.sha)) {
if (commit.pullRequest) {
@@ -284,14 +296,17 @@
@@ -284,14 +304,17 @@
}
if (this.lastReleaseSha && this.lastReleaseSha === commit.sha) {
this.logger.info(`Using configured lastReleaseSha ${this.lastReleaseSha} as last commit.`);
@@ -153,7 +168,7 @@
break;
}
commits.push({
@@ -301,16 +316,18 @@
@@ -301,16 +324,18 @@
pullRequest: commit.pullRequest,
});
}
+10 -4
View File
@@ -45,13 +45,19 @@ const operationResultKey = (contract) => {
return evidence ? "normalized_result" : "result";
};
/** Expand compact session metadata once at the verifier boundary. */
/**
* Expand compact session metadata once at the verifier boundary. The caller
* has validated the status against the binary_session contract, so a summary
* carries summary candidates without capability sets; complete detail must
* still expand every candidate's capabilities.
*/
export const expandMcpSessionStatus = (status) => ({
...status,
capabilities: expandProviderCapabilities(status.capabilities),
analysis_provider_candidates: expandProviderCandidates(
status.analysis_provider_candidates,
),
analysis_provider_candidates:
status.detail === "summary"
? status.analysis_provider_candidates
: expandProviderCandidates(status.analysis_provider_candidates),
tool_availability: expandToolAvailability(status.tool_availability),
});
+2 -2
View File
@@ -10,7 +10,7 @@ import {
toolContract,
} from "../dist/contracts/toolContracts.js";
import { advertisedInputJsonSchema } from "../dist/contracts/toolSchemaMetadata.js";
import { presentAdvertisedInputSchema } from "../dist/contracts/compactInputSchema.js";
import { presentAdvertisedInputProfile } from "../dist/contracts/inputSchemaPresentation.js";
import { compactPresentationForMcpInputSchemaProfile } from "../dist/config/mcpInputSchemaProfile.js";
import { createVerifierCommands } from "./lib/verify-commands.mjs";
import { withVerifierCleanup } from "./lib/verifier-cleanup.mjs";
@@ -186,7 +186,7 @@ await withVerifierCleanup(async () => {
);
assert.deepEqual(
tool.parameters,
presentAdvertisedInputSchema(
presentAdvertisedInputProfile(
advertisedInputJsonSchema(
toolContract(tool.name.slice(prefix.length)),
),
+32 -28
View File
@@ -66,6 +66,7 @@ import {
hasBinaryPlistSignature,
MAX_XML_PLIST_PROBE_BYTES,
} from "../domain/apple/appleArchiveFormat.js";
import { hasDiskImageSignature } from "../domain/apple/diskImageFormat.js";
import { hasGodotPckHeader } from "../domain/godotPckFormat.js";
import {
hasZipSignature,
@@ -638,11 +639,10 @@ const detectArtifactFormat = async (
const { handle, initial } = source;
throwIfTargetResolutionCancelled(signal);
const lower = path.toLowerCase();
const magic = Buffer.alloc(4096);
const observed = await handle.read(magic, 0, magic.length, 0);
throwIfTargetResolutionCancelled(signal);
// Fill the prefix so a short host read cannot hide a signature.
const magic = Buffer.from(await readHandleRange(handle, 0, 4096, signal));
if (
observed.bytesRead >= 8 &&
magic.length >= 8 &&
magic.subarray(0, 8).equals(Buffer.from([0, 0x61, 0x73, 0x6d, 1, 0, 0, 0]))
)
throw new BinaryTargetError(
@@ -650,39 +650,32 @@ const detectArtifactFormat = async (
"WebAssembly v1 artifacts are not supported by the native opening route",
{ constraint: "wasm_requires_static_inspection" },
);
if (hasZipSignature(magic.subarray(0, observed.bytesRead)))
return zipRootPackageFormat(lower);
if (hasGodotPckHeader(magic.subarray(0, observed.bytesRead)))
return "godot-pck";
if (hasZipSignature(magic)) return zipRootPackageFormat(lower);
if (hasGodotPckHeader(magic)) return "godot-pck";
// Binary plists, including keyed archives, often use other extensions.
if (hasBinaryPlistSignature(magic.subarray(0, observed.bytesRead)))
return "plist";
if (hasBinaryPlistSignature(magic)) return "plist";
const named = namedArtifactFormat(lower);
if (named !== undefined) return named;
if (
lower.endsWith(".pkg") &&
observed.bytesRead >= 4 &&
magic.length >= 4 &&
magic.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
if (lower.endsWith(".dmg")) {
throwIfTargetResolutionCancelled(signal);
const size = initial.size;
if (size >= 512) {
const trailer = Buffer.alloc(4);
const read = await handle.read(trailer, 0, trailer.length, size - 512);
throwIfTargetResolutionCancelled(signal);
if (read.bytesRead === 4 && trailer.equals(Buffer.from("koly")))
return "dmg";
}
}
if (
lower.endsWith(".dmg") &&
(await hasDiskImageSignature(initial.size, (position, length) =>
readHandleRange(handle, position, length, signal),
))
)
return "dmg";
// XML property lists can use archive-specific suffixes such as .archive.
// The named-format check above keeps .js, .map, and .asar classifications.
const xmlProbe = createXmlPlistSignatureProbe();
let xmlState = xmlProbe.write(magic.subarray(0, observed.bytesRead));
let xmlState = xmlProbe.write(magic);
if (xmlState === "matched") return "plist";
const chunk = Buffer.alloc(XML_PLIST_PROBE_CHUNK_BYTES);
let bytesRead = observed.bytesRead;
let bytesRead = magic.length;
while (xmlState === "pending" && bytesRead < MAX_XML_PLIST_PROBE_BYTES) {
throwIfTargetResolutionCancelled(signal);
const read = await handle.read(
@@ -747,7 +740,7 @@ const claimsLeadingExecutableHeader = async (
size,
);
/** Read one bounded range, honoring cancellation around the host read. */
/** Fill one bounded range, honoring cancellation; a short result means end of file. */
const readHandleRange = async (
handle: FileHandle,
position: number,
@@ -756,9 +749,20 @@ const readHandleRange = async (
): Promise<Uint8Array> => {
throwIfTargetResolutionCancelled(signal);
const bytes = Buffer.alloc(length);
const observed = await handle.read(bytes, 0, length, position);
throwIfTargetResolutionCancelled(signal);
return bytes.subarray(0, observed.bytesRead);
let filled = 0;
while (filled < length) {
if (filled > 0) throwIfTargetResolutionCancelled(signal);
const observed = await handle.read(
bytes,
filled,
length - filled,
position + filled,
);
throwIfTargetResolutionCancelled(signal);
if (observed.bytesRead === 0) break;
filled += observed.bytesRead;
}
return bytes.subarray(0, filled);
};
/** Read exactly `size` bytes by offset; a short read means the file changed or ended early. */
@@ -9,6 +9,7 @@ import { TOOL_CONTRACTS } from "../contracts/toolContracts.js";
import {
expandToolAvailability,
toolAvailabilityInventory,
toolAvailabilitySummaryInventory,
type ToolAvailability,
} from "../contracts/toolOutputSchemaPrimitives.js";
@@ -885,3 +886,44 @@ it("keeps independent native sessions discoverable without an unrelated binary t
{ available: false, reason: "target_required" },
);
});
describe("summary tool availability", () => {
it("summarizes every tool's current availability without static catalog flags", () => {
const full = buildCapabilityInventory(status(), enabledPolicy);
const withOverride = full.map((entry, index) =>
index === 1
? {
...entry,
available: false as const,
reason: "client_capability_missing" as const,
remediation: "Use an MCP client with roots support.",
client_requirements: {
required: ["roots" as const],
optional: [],
missing_required: ["roots" as const],
missing_optional: [],
},
}
: entry,
);
const complete = compactToolAvailability(withOverride);
const summary = compactToolAvailability(withOverride, "summary");
expect(toolAvailabilitySummaryInventory.parse(summary)).toEqual(summary);
expect(toolAvailabilityInventory.safeParse(summary).success).toBe(false);
expect(summary.groups.length).toBeLessThan(complete.groups.length);
expect(JSON.stringify(summary).length).toBeLessThan(
JSON.stringify(complete).length / 2,
);
for (const group of summary.groups)
for (const field of ["surface", "effects", "annotations"])
expect(Object.hasOwn(group, field)).toBe(false);
const byName = (left: { name: string }, right: { name: string }) =>
left.name.localeCompare(right.name);
expect(expandToolAvailability(summary).toSorted(byName)).toEqual(
withOverride
.map(({ surface: _s, effects: _e, annotations: _a, ...entry }) => entry)
.toSorted(byName),
);
});
});
+38 -19
View File
@@ -6,6 +6,7 @@ import { NATIVE_DISPATCH_METADATA_SOURCES } from "./native/NativeDispatchMetadat
import { NATIVE_UI_ACTION_OPERATIONS } from "./native/NativeUiActionTrace.js";
import { NATIVE_CALL_ROUTE_SOURCES } from "./native/NativeCallRoutes.js";
import { z } from "zod";
import type { BinarySessionDetail } from "../contracts/sessionStatusContract.js";
import { sqliteDatabaseCapability } from "./sqlite/SqliteDatabaseCapability.js";
import type {
OptionalProviderLoadFailures,
@@ -16,11 +17,13 @@ import {
expandProviderCapabilities,
providerCapabilitySet,
toolAvailabilityInventory,
toolAvailabilitySummaryInventory,
TOOL_ANNOTATION_NAMES,
TOOL_EFFECT_NAMES,
type ClientFeatureAvailability,
type ProviderCapability,
type ToolAvailabilityInventory,
type ToolAvailabilitySummaryInventory,
type ToolAvailability,
type ToolAvailabilityMode,
type ToolUnavailabilityReason,
@@ -323,10 +326,6 @@ export const buildCapabilityInventory = (
};
type ClientRequirements = ToolAvailability["client_requirements"];
type MutableAvailabilityGroup = Omit<
ToolAvailabilityInventory["groups"][number],
"names"
> & { names: string[] };
const EMPTY_CLIENT_REQUIREMENTS: ClientRequirements = {
required: [],
@@ -335,23 +334,29 @@ const EMPTY_CLIENT_REQUIREMENTS: ClientRequirements = {
missing_optional: [],
};
/** Compact repeated metadata while retaining every operation's current availability inline. */
export const compactToolAvailability = (
/**
* Group every operation's current availability inline. Complete detail also
* groups by the static surface, effect, and annotation flags; summary omits
* those flags so tools that share a status collapse into one group.
*/
export function compactToolAvailability(
availability: readonly ToolAvailability[],
): ToolAvailabilityInventory => {
detail?: "complete",
): ToolAvailabilityInventory;
export function compactToolAvailability(
availability: readonly ToolAvailability[],
detail: BinarySessionDetail,
): ToolAvailabilityInventory | ToolAvailabilitySummaryInventory;
export function compactToolAvailability(
availability: readonly ToolAvailability[],
detail: BinarySessionDetail = "complete",
): ToolAvailabilityInventory | ToolAvailabilitySummaryInventory {
const clientRequirements =
availability[0]?.client_requirements ?? EMPTY_CLIENT_REQUIREMENTS;
const groups = new Map<string, MutableAvailabilityGroup>();
const groups = new Map<string, { names: string[] }>();
for (const entry of availability) {
const effects = TOOL_EFFECT_NAMES.filter((name) => entry.effects[name]);
const annotations = TOOL_ANNOTATION_NAMES.filter(
(name) => entry.annotations[name],
);
const groupFacts = {
surface: entry.surface,
effects,
annotations,
const statusFacts = {
available: entry.available,
reason: entry.reason,
remediation: entry.remediation,
@@ -364,17 +369,31 @@ export const compactToolAvailability = (
? {}
: { client_requirements: entry.client_requirements }),
};
const groupFacts =
detail === "summary"
? statusFacts
: {
surface: entry.surface,
effects: TOOL_EFFECT_NAMES.filter((name) => entry.effects[name]),
annotations: TOOL_ANNOTATION_NAMES.filter(
(name) => entry.annotations[name],
),
...statusFacts,
};
const key = JSON.stringify(groupFacts);
const existing = groups.get(key);
if (existing) existing.names.push(entry.name);
else groups.set(key, { ...groupFacts, names: [entry.name] });
}
return toolAvailabilityInventory.parse({
const inventory = {
client_requirements: clientRequirements,
groups: [...groups.values()],
});
};
};
return detail === "summary"
? toolAvailabilitySummaryInventory.parse(inventory)
: toolAvailabilityInventory.parse(inventory);
}
const availabilityFor = (context: AvailabilityContext): Availability => {
// These built-in workflows accept a caller-selected endpoint; they do not load the optional passive provider.
@@ -0,0 +1,521 @@
import { createHash } from "node:crypto";
import { describe, expect, it } from "vitest";
import {
SCENARIO_ORIGIN,
scenarioCaptureEvidence,
scenarioCaptureFixture,
} from "../../../tests/fixtures/browserScenarioCapture.js";
import { inspectAnalysisView } from "./AnalysisViewService.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { sanitizeBrowserUrl } from "../../domain/browserObservation.js";
import type { BrowserScenarioCapture } from "../../domain/browserScenarioCapture.js";
import type { AnalysisViewRequest } from "../../domain/analysisView/analysisView.js";
import { projectBrowserScenarioView } from "../../domain/analysisView/browserScenarioView.js";
const tree = (text: string) => ({
state: "captured" as const,
value: {
sha256: createHash("sha256").update(text).digest("hex"),
bytes: Buffer.byteLength(text),
text,
},
});
/** Three steps: navigate to /quiz, capture its tree, then fail on a missing control. */
const quizCapture = (
thirdStep: "failed" | "completed" = "failed",
): BrowserScenarioCapture => {
const base = scenarioCaptureFixture();
const [initial, browse] = base.steps;
if (initial === undefined || browse === undefined)
throw new Error("Fixture lacks steps");
const quiz = sanitizeBrowserUrl(`${SCENARIO_ORIGIN}/quiz`);
return {
...base,
scenario: { ...base.scenario, action_count: 2 },
steps: [
{
...initial,
artifacts: {
...initial.artifacts,
url: { state: "captured", value: initial.after_url },
},
},
{
...browse,
after_url: quiz,
artifacts: {
...browse.artifacts,
accessibility: tree('- heading "Question 1"'),
screenshot: { state: "missing", reason: "page closed" },
},
},
{
...browse,
step_index: 2,
step_id: "answer",
action: "click",
before_url: quiz,
after_url: quiz,
...(thirdStep === "failed"
? { status: "failed" as const, error: "locator not found" }
: { status: "completed" as const, error: null }),
},
],
};
};
const view = async (
capture: BrowserScenarioCapture,
request: AnalysisViewRequest,
) => {
const projected = await inspectAnalysisView({
source: { kind: "inline", evidence: scenarioCaptureEvidence(capture) },
view: request,
});
if (!projected.ok) throw projected.error;
return projected.value.normalized_result as Record<string, unknown>;
};
describe("browser scenario analysis views", () => {
it("counts inherited Evidence limitations alongside capture-local caveats", async () => {
const capture = scenarioCaptureFixture();
expect(capture.limitations).toEqual([]);
const summary = await view(capture, { kind: "summary" });
expect(summary).toMatchObject({
limitations: [
"Artifact identity is unavailable for this observation.",
"Browser host platform and architecture were not observed.",
],
summary: { limitation_count: 2 },
});
});
it("summarizes steps compactly and focuses the first failing step's nearest captured tree", async () => {
const summary = await view(quizCapture(), { kind: "summary" });
expect(summary).toMatchObject({
kind: "summary",
parent_operation: "capture_browser_scenario",
artifact: null,
coverage: { status: "complete-within-view", exhausted: true },
});
const { steps, final_accessibility, network, events } =
summary.summary as Record<string, unknown> & {
steps: Record<string, unknown>[];
};
// before_url repeats only across a discontinuity; not_requested sections are omitted.
expect(steps[0]).toHaveProperty("before_url");
expect(steps[1]).not.toHaveProperty("before_url");
expect(steps[2]).not.toHaveProperty("before_url");
expect(steps[0]?.sections).toEqual({
captured: ["url"],
missing: [],
truncated: [],
});
expect(steps[1]?.sections).toEqual({
captured: ["accessibility"],
missing: ["screenshot"],
truncated: [],
});
expect(steps[2]).toMatchObject({
status: "failed",
error: "locator not found",
after_url: { url: `${SCENARIO_ORIGIN}/quiz` },
});
expect(final_accessibility).toEqual({
focus_step_index: 2,
focus: "first-unsuccessful-step",
step_index: 1,
accessibility: tree('- heading "Question 1"'),
});
expect(network).toEqual({
selected: true,
transactions: 8,
without_request: 0,
ambiguous_transactions: 0,
failed_transactions: 1,
by_origin: [{ origin: SCENARIO_ORIGIN, transactions: 8 }],
by_resource_type: [
{ resource_type: "document", transactions: 1 },
{ resource_type: "fetch", transactions: 7 },
],
by_status_class: [
{ status_class: "2xx", transactions: 7 },
{ status_class: "failed", transactions: 1 },
],
});
expect(events).toMatchObject({ retained: 24, dropped: 0 });
});
it("reports the last step's own state when no step captured a tree", async () => {
const capture = scenarioCaptureFixture();
expect(await view(capture, { kind: "summary" })).toMatchObject({
summary: {
final_accessibility: {
focus_step_index: 1,
focus: "last-step",
step_index: null,
accessibility: { state: "not_requested" },
},
},
});
});
});
describe("browser scenario analysis view pages", () => {
it("pages exact steps with only the selected captured sections", async () => {
const page = await view(quizCapture("completed"), {
kind: "browser-scenario",
collection: "steps",
step_indices: [1, 2],
artifacts: ["accessibility"],
offset: 0,
limit: 1,
});
expect(page).toMatchObject({
kind: "browser-scenario",
source_completeness: { status: "complete" },
source_events: { retained: 24, dropped: 0 },
coverage: { status: "page", examined: 1, total: 2, next_offset: 1 },
items: {
collection: "steps",
rows: [
{
step_index: 1,
artifacts: { accessibility: tree('- heading "Question 1"') },
omitted_artifacts: [{ section: "screenshot", state: "missing" }],
},
],
},
});
});
it("filters events by step, kind and transaction, keeping original item indices", async () => {
const capture = scenarioCaptureFixture();
const responses = (await view(capture, {
kind: "browser-scenario",
collection: "events",
event_kinds: ["response", "request-failed"],
transaction_ids: ["request-2", "request-8"],
offset: 0,
limit: 64,
})) as { items: { rows: { item_index: number; event: unknown }[] } };
expect(
responses.items.rows.map(({ item_index, event }) => ({
item_index,
event,
})),
).toEqual(
capture.events.items.flatMap((event, item_index) =>
(event.kind === "response" || event.kind === "request-failed") &&
["request-2", "request-8"].includes(event.transaction_id)
? [{ item_index, event }]
: [],
),
);
expect(responses.items.rows).toHaveLength(2);
for (const empty of [
{ step_indices: [] },
{ event_kinds: [] },
{ transaction_ids: [] },
{ step_indices: [0] },
])
expect(
await view(capture, {
kind: "browser-scenario",
collection: "events",
offset: 0,
limit: 64,
...empty,
}),
).toMatchObject({
items: { rows: [] },
coverage: { status: "empty", total: 0, exhausted: true },
});
});
it("states dropped events and refuses views of other parents", async () => {
const base = scenarioCaptureFixture();
const dropped = {
...base,
events: {
...base.events,
dropped: 3,
omissions: {
...base.events.omissions,
by_kind: [
{
kind: "console" as const,
count: 3,
reasons: ["max-events" as const],
},
],
},
},
completeness: {
status: "truncated" as const,
equality_eligible: false as const,
missing_sections: [],
truncated_sections: ["events" as const],
},
};
const summary = await view(dropped, { kind: "summary" });
expect(summary.unknowns).toContain(
"3 events were dropped by capture budgets; counts and pages cover retained events only.",
);
expect(summary).toMatchObject({
summary: {
events: {
dropped: 3,
omissions: {
by_kind: [{ kind: "console", count: 3 }],
network_transactions: 0,
},
},
},
});
const refused = await inspectAnalysisView({
source: { kind: "inline", evidence: scenarioCaptureEvidence() },
view: { kind: "go", collection: "dependencies" },
});
if (refused.ok) throw new Error("Expected an unsupported view refusal");
expect(projectAnalysisError(refused.error)).toMatchObject({
code: "invalid_request",
});
});
});
describe("browser scenario retained boundaries", () => {
it("counts transactions whose request event was omitted and keeps source omissions on an exhausted page", async () => {
const base = scenarioCaptureFixture();
const omitted = (
event: BrowserScenarioCapture["events"]["items"][number],
) =>
event.kind === "request" ||
(event.kind === "network-content" && event.phase === "request");
const removed = base.events.items.filter(omitted);
const items = base.events.items.filter((event) => !omitted(event));
const capture: BrowserScenarioCapture = {
...base,
events: {
...base.events,
items,
retained: items.length,
dropped: removed.length,
omissions: {
...base.events.omissions,
by_kind: [
{ kind: "request", count: 8, reasons: ["max-events"] },
{ kind: "network-content", count: 1, reasons: ["max-events"] },
],
},
},
completeness: {
status: "truncated",
equality_eligible: false,
missing_sections: [],
truncated_sections: ["events"],
},
};
expect(await view(capture, { kind: "summary" })).toMatchObject({
summary: {
network: {
transactions: 8,
without_request: 8,
failed_transactions: 1,
},
},
});
expect(
await view(capture, {
kind: "browser-scenario",
collection: "transactions",
transaction_ids: ["request-8"],
event_kinds: [],
offset: 0,
limit: 1,
}),
).toMatchObject({
source_completeness: { status: "truncated" },
source_events: { dropped: 9 },
coverage: { total: 1, exhausted: true },
items: {
rows: [
{
transaction_id: "request-8",
events: [],
omitted_events: [{ kind: "request-failed" }],
},
],
},
});
});
it("keeps network bodies opt-in while retaining each transaction's event identities", async () => {
const capture = scenarioCaptureFixture();
const request = {
kind: "browser-scenario" as const,
collection: "transactions" as const,
transaction_ids: ["request-2"],
offset: 0,
limit: 1,
};
// Parse defaults at the public boundary rather than supplying a manually normalized request.
const page = await inspectAnalysisView({
source: { kind: "inline", evidence: scenarioCaptureEvidence(capture) },
view: request,
});
if (!page.ok) throw page.error;
const result = page.value.normalized_result;
expect(result).toMatchObject({
items: {
collection: "transactions",
rows: [{ transaction_id: "request-2" }],
},
});
expect(JSON.stringify(result)).not.toContain('"content":');
const selected = await view(capture, {
...request,
event_kinds: ["network-content"],
});
expect(JSON.stringify(selected)).toContain('"content":');
expect(selected).toMatchObject({
items: {
rows: [
{
omitted_events: expect.arrayContaining([
{ item_index: expect.any(Number), kind: "request" },
]),
},
],
},
});
});
it("distinguishes unrequested and legacy-unknown network observation from an observed empty inventory", () => {
const base = scenarioCaptureFixture([], {
request_body: false,
response_body: false,
header_values: false,
});
const project = (capture: unknown) => {
const result = projectBrowserScenarioView(
{
evidenceId: `ev:${"a".repeat(64)}`,
operation: "capture_browser_scenario",
normalizedResult: capture,
limitations: [],
},
{ kind: "summary" },
);
if (!result.ok) throw result.error;
return result.value;
};
expect(project(base)).toMatchObject({
summary: { network: { selected: true, transactions: 0 } },
});
expect(
project({ ...base, scenario: { ...base.scenario, event_families: [] } }),
).toMatchObject({
summary: { network: { selected: false, transactions: null } },
});
const { event_families: _families, ...legacyScenario } = base.scenario;
expect(project({ ...base, scenario: legacyScenario })).toMatchObject({
summary: { network: { selected: null, transactions: null } },
});
});
});
describe("browser scenario text and selection boundaries", () => {
it("pages exact Unicode text with the full artifact digest and reports unavailable text explicitly", async () => {
const capture = quizCapture("completed");
const step = capture.steps[1]!;
step.artifacts.accessibility = tree("A💡B\nC");
const request = {
kind: "browser-scenario" as const,
collection: "text" as const,
step_index: 1,
artifact: "accessibility" as const,
offset: 1,
limit: 2,
};
expect(await view(capture, request)).toMatchObject({
items: {
state: "captured",
text: "💡",
sha256: tree("A💡B\nC").value.sha256,
bytes: Buffer.byteLength("A💡B\nC"),
},
coverage: { examined: 2, total: 6, next_offset: 3 },
});
expect(await view(capture, { ...request, offset: 99 })).toMatchObject({
items: { text: "" },
coverage: { status: "empty", total: 6, exhausted: true },
});
expect(await view(capture, { ...request, step_index: 0 })).toMatchObject({
items: { state: "not_requested", text: null, sha256: null, bytes: null },
});
});
it("rejects irrelevant selection fields and nonexistent text steps with typed diagnostics", async () => {
for (const request of [
{ kind: "browser-scenario", collection: "events", artifacts: [] },
{
kind: "browser-scenario",
collection: "text",
step_index: 99,
artifact: "dom",
},
]) {
const result = await inspectAnalysisView({
source: { kind: "inline", evidence: scenarioCaptureEvidence() },
view: request,
});
if (result.ok)
throw new Error("Expected the invalid view to be rejected");
expect(projectAnalysisError(result.error)).toMatchObject({
code: "invalid_request",
});
}
});
it("keeps source truncation facts when a text page has no retained payload", async () => {
const capture = quizCapture("completed");
const completeness = {
status: "truncated" as const,
equality_eligible: false as const,
missing_sections: [],
truncated_sections: ["accessibility" as const],
};
capture.steps[1]!.artifacts.accessibility = {
state: "truncated",
observed: 1000,
retained: 0,
reason: "capture budget",
};
capture.steps[1]!.completeness = completeness;
capture.completeness = completeness;
expect(
await view(capture, {
kind: "browser-scenario",
collection: "text",
step_index: 1,
artifact: "accessibility",
offset: 0,
limit: 20,
}),
).toMatchObject({
source_completeness: { status: "truncated" },
items: {
state: "truncated",
observed: 1000,
retained: 0,
reason: "capture budget",
text: null,
},
coverage: { status: "empty", exhausted: true },
});
});
});
@@ -0,0 +1,320 @@
import { expect, it } from "vitest";
import { createEvidence } from "../../domain/evidence.js";
import {
analysisViewResultSchema,
inspectAnalysisViewInputSchema,
} from "../../domain/analysisView/analysisView.js";
import { projectNativeLoadImageView } from "../../domain/analysisView/nativeLoadImageView.js";
import type { NativeLoadImage } from "../../domain/native/nativeLoadImage.js";
import {
nativeDosLoadImageFixture,
nativeLoadImageObservationFixture,
} from "../../../tests/fixtures/nativeLoadImage.js";
import { inspectAnalysisView } from "./AnalysisViewService.js";
const source = (): Extract<NativeLoadImage, { status: "unsupported" }> => ({
status: "unsupported",
reason:
"ELF load observations have no independent verification in this fixture.",
observations: {
...nativeLoadImageObservationFixture(),
executable_format: "Executable and Linking Format (ELF)",
relocations: [
{
address: "ram:0x0010",
segment: null,
offset: null,
status: "APPLIED",
type: 8,
values: [16],
original_bytes_hex: "0000",
memory_bytes_hex: "1000",
},
{
address: "ram:0x0010",
segment: null,
offset: null,
status: "UNSUPPORTED",
type: 9,
values: [],
original_bytes_hex: null,
memory_bytes_hex: null,
},
],
},
limitations: [
"Load observations do not establish runtime relocation behavior.",
],
});
const evidence = (report: NativeLoadImage) =>
createEvidence(
{ path: "/selected/fixture", sha256: "a".repeat(64), format: "elf" },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "inspect_native_load_image",
parameters: {},
result: report,
limitations: ["Provider observation remains historical."],
},
{ immutable: true },
);
const view = async (report: NativeLoadImage, request: unknown) => {
const result = await inspectAnalysisView({
source: { kind: "inline", evidence: evidence(report) },
view: request,
});
if (!result.ok) throw result.error;
return analysisViewResultSchema.parse(result.value.normalized_result);
};
it("summarizes measured ELF facts without upgrading unsupported verification", async () => {
const report = source();
const parent = evidence(report);
const summary = await view(report, { kind: "summary" });
expect(summary).toMatchObject({
parent_evidence_id: parent.evidence_id,
parent_operation: "inspect_native_load_image",
source_provider: parent.provider,
source_analysis_profile_digest: null,
source_parameters: {},
artifact: { path: "/selected/fixture", sha256: "a".repeat(64) },
summary: {
verification: { status: "unsupported", reason: report.reason },
observation: {
executable_format: "Executable and Linking Format (ELF)",
},
counts: {
relocations: 2,
checks: null,
sections: null,
metadata_recovery: null,
},
},
limitations: expect.arrayContaining([
...report.limitations,
"Provider observation remains historical.",
]),
unknowns: expect.arrayContaining([report.reason]),
});
expect(JSON.stringify(summary)).not.toContain("memory_bytes_hex");
});
it("preserves same-address alternatives, padding, nulls and byte facts over pages", async () => {
const report = source();
const records = [];
for (let offset = 0; offset < 2; offset++) {
const page = await view(report, {
kind: "native-load-image",
collection: "relocations",
fields: [
"address",
"segment",
"offset",
"status",
"type",
"values",
"original_bytes_hex",
"memory_bytes_hex",
],
offset,
limit: 1,
});
if (page.kind !== "native-load-image") throw new Error("Expected page");
expect(page.verification_status).toBe("unsupported");
expect(
analysisViewResultSchema.safeParse({
...page,
items: {
...page.items,
rows: null,
},
}).success,
).toBe(false);
expect(page.items.rows?.[0]?.source_index).toBe(offset);
expect(page.coverage).toMatchObject({
total: 2,
next_offset: offset === 0 ? 1 : null,
exhausted: offset === 1,
});
records.push(...(page.items.rows ?? []).map((row) => row.record));
}
expect(records).toEqual(report.observations.relocations);
});
it("filters indices before paging and projects only selected fields", async () => {
const page = await view(source(), {
kind: "native-load-image",
collection: "relocations",
indices: [1, 400, 0, 1],
offset: 1,
fields: ["address", "status", "status"],
limit: 1,
});
expect(page).toMatchObject({
items: {
collection: "relocations",
rows: [
{
source_index: 1,
record: { address: "ram:0x0010", status: "UNSUPPORTED" },
},
],
},
coverage: { examined: 1, total: 2, next_offset: null, exhausted: true },
});
if (page.kind !== "native-load-image") throw new Error("Expected page");
expect(page.items.rows?.[0]?.record).toEqual({
address: "ram:0x0010",
status: "UNSUPPORTED",
});
});
it.each(["checks", "sections"])(
"keeps unrecorded or inapplicable %s null rather than inventing empty observations",
async (collection) => {
expect(
await view(source(), { kind: "native-load-image", collection }),
).toMatchObject({
items: { collection, rows: null },
unknowns: expect.arrayContaining([
"Null collection counts or rows mean not recorded or not applicable, not observed absence.",
]),
});
},
);
it("distinguishes an explicitly recorded empty metadata collection and exhausted selection", async () => {
const report = source();
report.observations.metadata_recovery = [];
expect(await view(report, { kind: "summary" })).toMatchObject({
summary: { counts: { metadata_recovery: 0 } },
});
for (const request of [
{ collection: "entry_context" },
{ collection: "relocations", indices: [] },
{ collection: "relocations", offset: 400 },
])
expect(
await view(report, { kind: "native-load-image", ...request }),
).toMatchObject({
items: { rows: [] },
coverage: { status: "empty", exhausted: true, next_offset: null },
});
});
it("preserves recorded DOS verification details and ordered mismatch checks", async () => {
const report = nativeDosLoadImageFixture();
const summary = await view(report, { kind: "summary" });
expect(summary).toMatchObject({
summary: {
verification: {
status: "verified",
format: "dos-mz",
target_sha256: report.target_sha256,
},
counts: { checks: report.checks.length },
},
});
const mismatch = {
...report,
status: "mismatch" as const,
checks: [
{
...report.checks[0]!,
matched: false,
expected: { original: 1 },
observed: null,
},
],
};
expect(
await view(mismatch, {
kind: "native-load-image",
collection: "checks",
fields: [
"name",
"matched",
"expected",
"observed",
"file_offset",
"address",
],
}),
).toMatchObject({
items: { rows: [{ source_index: 0, record: mismatch.checks[0] }] },
});
expect(await view(mismatch, { kind: "summary" })).toMatchObject({
summary: { verification: { status: "mismatch" } },
});
});
it("keeps byte and potentially large comparison payloads opt-in while retaining full source facts", async () => {
const relocations = await view(source(), {
kind: "native-load-image",
collection: "relocations",
});
if (relocations.kind !== "native-load-image")
throw new Error("Expected page");
expect(relocations.items.rows?.[0]?.record).toEqual({
address: "ram:0x0010",
segment: null,
offset: null,
status: "APPLIED",
type: 8,
values: [16],
});
const report = nativeDosLoadImageFixture();
const body = { complete_comparison: "retained proof body".repeat(128) };
report.checks[0]!.expected = body;
const checks = await view(report, {
kind: "native-load-image",
collection: "checks",
});
if (checks.kind !== "native-load-image") throw new Error("Expected page");
expect(checks.items.rows?.[0]?.record).toEqual({
name: "original_file_sha256",
matched: true,
file_offset: null,
address: null,
});
expect(
await view(report, {
kind: "native-load-image",
collection: "checks",
fields: ["expected"],
}),
).toMatchObject({
items: { rows: [{ source_index: 0, record: { expected: body } }] },
});
expect(report.checks[0]!.expected).toEqual(body);
});
it("rejects collection-inapplicable fields and malformed sibling facts even for an empty selection", async () => {
expect(
inspectAnalysisViewInputSchema.safeParse({
source: { kind: "inline", evidence: evidence(source()) },
view: {
kind: "native-load-image",
collection: "entry_points",
fields: ["original_bytes_hex"],
},
}).success,
).toBe(false);
const report = source();
const parent = {
evidenceId: "ev_" + "b".repeat(64),
operation: "inspect_native_load_image",
normalizedResult: report,
limitations: [],
};
const request = {
kind: "native-load-image" as const,
collection: "relocations" as const,
indices: [],
offset: 0,
limit: 1,
};
expect(projectNativeLoadImageView(parent, request).ok).toBe(true);
Reflect.set(report.observations.relocations[1]!, "status", "");
expect(projectNativeLoadImageView(parent, request).ok).toBe(false);
});
+7 -1
View File
@@ -36,6 +36,7 @@ import { InvestigationRecords } from "../investigation/InvestigationRecords.js";
import { recordEvidenceSources } from "../investigation/recordEvidenceSources.js";
import { BinarySessionRecords } from "./BinarySessionRecords.js";
import { binarySessionStatus } from "./BinarySessionStatus.js";
import type { BinarySessionDetail } from "../../contracts/sessionStatusContract.js";
import {
resolveSessionOpen,
resolveSessionTarget,
@@ -1024,7 +1025,12 @@ export class BinarySession
}
/** Describe the active binary session. */
status(options: { readonly signal?: AbortSignal } = {}): Promise<JsonValue> {
status(
options: {
readonly signal?: AbortSignal;
readonly detail?: BinarySessionDetail;
} = {},
): Promise<JsonValue> {
const resourceRun = this.#activeBundleResources;
const run =
this.#active ??
+6 -1
View File
@@ -8,6 +8,7 @@ import type { JsonValue } from "../../domain/jsonValue.js";
import type { AnalysisSnapshot } from "../../domain/analysisSnapshot.js";
import type { AnalysisProfileCommitment } from "../../domain/analysisProfile.js";
import type { AnalysisProviderSelector } from "../../contracts/providerSelection.js";
import type { BinarySessionDetail } from "../../contracts/sessionStatusContract.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import type { Result } from "../../domain/result.js";
import type { EvidenceIntegrityError } from "../../domain/evidenceErrors.js";
@@ -93,7 +94,11 @@ export interface BinarySessionPort
overwrite: boolean,
options?: Pick<ExecutionOptions, "progress">,
): Promise<Result<SavedAnalysisSnapshot, AnalysisError>>;
status(options?: { readonly signal?: AbortSignal }): Promise<JsonValue>;
/** Default complete detail; summary omits provider diagnostics and candidate capabilities. */
status(options?: {
readonly signal?: AbortSignal;
readonly detail?: BinarySessionDetail;
}): Promise<JsonValue>;
activeTarget(): BinaryTarget | undefined;
/** Read a snapshot with failed descriptors retained for session cleanup retry. */
readAnalysisSnapshot(
@@ -5,9 +5,12 @@ import type {
CapabilityDescriptor,
ProviderIdentity,
} from "../AnalysisProvider.js";
import { z } from "zod";
import {
analysisProviderCandidate,
analysisProviderCandidateSummary,
expandProviderCapabilities,
sessionProvider,
sessionProviderFacts,
type ProviderCapabilitySet,
} from "../../contracts/toolOutputSchemaPrimitives.js";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
@@ -23,9 +26,13 @@ import {
nativeMacOSCapabilities,
} from "../../native/NativeMacOSProviderMetadata.js";
const projectedStatusSchema = sessionProvider.pick({
capabilities: true,
analysis_provider_candidates: true,
const projectedStatusSchema = z.object({
capabilities: sessionProviderFacts.shape.capabilities,
analysis_provider_candidates: z.array(analysisProviderCandidate),
});
const summaryStatusSchema = z.object({
capabilities: sessionProviderFacts.shape.capabilities,
analysis_provider_candidates: z.array(analysisProviderCandidateSummary),
});
const providerA: ProviderIdentity = {
@@ -126,6 +133,25 @@ const project = async (
{ readonly reason: string }
> = new Map(),
target?: BinaryTarget,
) =>
projectedStatusSchema.parse(
await projectStatus(routeCapabilities, candidates, {
runtimeUnavailability,
...(target === undefined ? {} : { target }),
}),
);
const projectStatus = async (
routeCapabilities: readonly CapabilityDescriptor[],
candidates: readonly AnalysisProviderCandidateStatus[],
options: {
readonly runtimeUnavailability?: ReadonlyMap<
string,
{ readonly reason: string }
>;
readonly target?: BinaryTarget;
readonly detail?: "summary" | "complete";
} = {},
) => {
const route: SessionProviderRoute = {
identity: providerA,
@@ -142,17 +168,16 @@ const project = async (
const router: Pick<SessionProviderRouter, "candidateStatuses"> = {
candidateStatuses: async () => candidates,
};
return projectedStatusSchema.parse(
await binarySessionStatus({
target,
route,
router,
runtimeUnavailability,
runId: undefined,
runtimeLineageSnapshots: [],
requestActivitySnapshots: [],
}),
);
return binarySessionStatus({
target: options.target,
route,
router,
runtimeUnavailability: options.runtimeUnavailability ?? new Map(),
runId: undefined,
runtimeLineageSnapshots: [],
requestActivitySnapshots: [],
...(options.detail === undefined ? {} : { detail: options.detail }),
});
};
const expandCanonicalCapabilities = (sets: readonly ProviderCapabilitySet[]) =>
@@ -444,3 +469,87 @@ it.each(["elf", "pe"] as const)(
]);
},
);
describe("binary session summary projection", () => {
it("keeps each candidate's status, code and reason in summary without diagnostics or capability sets", async () => {
const unavailable: AnalysisProviderCandidateStatus = {
...candidate(providerB, otherCapabilities),
availability: {
status: "unavailable",
code: "not_configured",
reason: "Set PROVIDER_B_HOME; see docs/provider-b.md.",
diagnostics: { checks: [{ name: "configuration", ok: false }] },
},
targetSupport: {
status: "unsupported",
code: "architecture_unsupported",
reason: "Provider B cannot select this slice.",
diagnostics: { architecture: "arm64" },
},
selected: false,
};
const pending: AnalysisProviderCandidateStatus = {
...candidate(providerA, selectedCapabilities),
availability: {
status: "unknown",
code: null,
reason: "Provider A startup is still pending.",
diagnostics: { startup: "pending" },
},
};
const candidates = [pending, unavailable];
const before = structuredClone(candidates);
const runtimeUnavailability = new Map([
["procedure_info", { reason: "Runtime operation failure" }],
]);
const complete = projectedStatusSchema.parse(
await projectStatus(selectedCapabilities, candidates, {
runtimeUnavailability,
}),
);
const raw = await projectStatus(selectedCapabilities, candidates, {
runtimeUnavailability,
detail: "summary",
});
const summary = summaryStatusSchema.parse(raw);
expect(candidates).toEqual(before);
expect(summary.capabilities).toEqual(complete.capabilities);
expect(summary.analysis_provider_candidates).toEqual([
{
provider: providerA,
availability: {
status: "unknown",
code: null,
reason: "Provider A startup is still pending.",
},
target_support: {
status: "unknown",
code: null,
reason: "No target is open.",
},
selected: true,
},
{
provider: providerB,
availability: {
status: "unavailable",
code: "not_configured",
reason: "Set PROVIDER_B_HOME; see docs/provider-b.md.",
},
target_support: {
status: "unsupported",
code: "architecture_unsupported",
reason: "Provider B cannot select this slice.",
},
selected: false,
},
]);
expect(JSON.stringify(raw)).not.toContain("diagnostics");
expect(
analysisProviderCandidate
.array()
.safeParse(summary.analysis_provider_candidates).success,
).toBe(false);
});
});
+41 -14
View File
@@ -19,6 +19,7 @@ import type {
SessionProviderRouter,
} from "./SessionProviderRouter.js";
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
import type { BinarySessionDetail } from "../../contracts/sessionStatusContract.js";
interface BinarySessionStatusInput {
readonly target: BinaryTarget | undefined;
@@ -35,6 +36,7 @@ interface BinarySessionStatusInput {
readonly requestActivitySnapshots: readonly ProviderRequestActivitySnapshot[];
readonly providerOperationHealth?: ProviderOperationHealth | null;
readonly cleanupPending?: boolean;
readonly detail?: BinarySessionDetail;
readonly signal?: AbortSignal;
}
@@ -51,6 +53,7 @@ export const binarySessionStatus = async ({
requestActivitySnapshots,
providerOperationHealth,
cleanupPending = false,
detail = "complete",
signal,
}: BinarySessionStatusInput): Promise<JsonValue> => {
const routeCapabilities = (
@@ -97,20 +100,31 @@ export const binarySessionStatus = async ({
const analysisProviderCandidates =
bundleResourceRoot !== undefined
? []
: (await router.candidateStatuses(route, signal)).map((candidate) => ({
provider: providerSummary(candidate.provider),
availability: structuredClone(candidate.availability),
target_support: structuredClone(candidate.targetSupport),
selected: candidate.selected,
capabilities: providerCapabilitySets(
candidate.capabilities
.slice()
.filter(({ operation }) => operation !== "inventory_artifact")
.sort((left, right) =>
compareUnicodeCodePoints(left.operation, right.operation),
),
),
}));
: (await router.candidateStatuses(route, signal)).map((candidate) =>
detail === "summary"
? {
provider: providerSummary(candidate.provider),
availability: providerStateSummary(candidate.availability),
target_support: providerStateSummary(candidate.targetSupport),
selected: candidate.selected,
}
: {
provider: providerSummary(candidate.provider),
availability: structuredClone(candidate.availability),
target_support: structuredClone(candidate.targetSupport),
selected: candidate.selected,
capabilities: providerCapabilitySets(
candidate.capabilities
.slice()
.filter(
({ operation }) => operation !== "inventory_artifact",
)
.sort((left, right) =>
compareUnicodeCodePoints(left.operation, right.operation),
),
),
},
);
const common = {
cleanup_pending: cleanupPending,
capabilities,
@@ -185,6 +199,19 @@ const requestActivityStatus = (
};
};
/** Keep a provider-owned status, code, and reason; diagnostics stay in complete detail. */
const providerStateSummary = <
State extends {
readonly status: string;
readonly code: string | null;
readonly reason: string | null;
},
>({
status,
code,
reason,
}: State) => ({ status, code, reason });
const providerSummary = ({
id,
name,
@@ -76,15 +76,15 @@ describe("environment manifests from capture_browser_scenario parents", () => {
kind: "inline",
evidence: scenarioCaptureEvidence(scenarioCaptureFixture()),
},
view: { kind: "summary" },
view: { kind: "dart-aot", collection: "package_uris" },
});
if (refused.ok) throw new Error("Expected the unsupported view refusal");
expect(projectAnalysisError(refused.error)).toMatchObject({
code: "unsupported_target",
message: expect.stringContaining(
"supports only the environment-manifest, environment-state and environment-coverage views",
),
code: "invalid_request",
});
expect(JSON.stringify(projectAnalysisError(refused.error))).toContain(
"supports summary and browser-scenario views",
);
});
it("serves and exports a scenario-built manifest from the same capture", () => {
@@ -2,7 +2,7 @@ import { expect, it } from "vitest";
import { createHash } from "node:crypto";
import { matchNativeFunctions } from "./NativeFunctionMatchingService.js";
import { nativeFunctionMatchingSchema } from "../../domain/native/functionMatching.js";
import { createEvidence } from "../../domain/evidence.js";
import { createEvidence, immutableEvidence } from "../../domain/evidence.js";
import { createAnalysisProfile } from "../../domain/analysisProfile.js";
import { functionDossierSchema } from "../../domain/hopperValues.js";
import { procedureInfoSchema } from "../../domain/native/procedureInfo.js";
@@ -791,6 +791,134 @@ it("cooperatively preserves the exact cancellation reason and rejects expansion
expect(large.error._tag).toBe("AnalysisResourceConstraintError");
});
const retainedInventories = (
rows: (side: "1" | "2") => readonly Record<string, unknown>[],
) => {
const [left, right] = (["1", "2"] as const).map((side) =>
immutableEvidence(
createEvidence(nativeMatchingTarget(side), nativeMatchingProvider, {
operation: "list_procedures",
parameters: {},
result: rows(side),
}),
),
);
const ledger = new Map([left!, right!].map((x) => [x.evidence_id, x]));
const reference = (record: { evidence_id: string }) => ({
kind: "retained-evidence" as const,
evidence_id: record.evidence_id,
});
return {
left: left!,
right: right!,
references: { left: reference(left!), right: reference(right!) },
lookup: (id: string) => ledger.get(id),
};
};
const ghidraRow = (side: "1" | "2", index: number) => ({
address: `0x${(0x400000 + index * 16 + (side === "2" ? 8 : 0)).toString(16)}`,
value: `subsystem_function_${index}`,
symbol: {
primary: true,
dynamic: false,
external: false,
type: "Function",
source: "imported",
},
procedure: { external: false, thunk: false, thunk_target: null },
});
it("charges a borrowed retained inventory for its emitted members, not its resident payload", async () => {
// One shared 40 KB string per row: ~40 KB resident, but a per-occurrence
// walk would charge 1000 copies of it.
const resident = "r".repeat(40_000);
const { left, right, references, lookup } = retainedInventories((side) =>
Array.from({ length: 1000 }, (_, i) => ({
...ghidraRow(side, i),
provider_note: resident,
})),
);
const borrowed = await matchNativeFunctions(references, undefined, lookup);
if (!borrowed.ok) throw borrowed.error;
expect(
nativeFunctionMatchingSchema.parse(
borrowed.value.evidence.normalized_result,
).summary.matched,
).toBe(1000);
// Inline Evidence is authenticated by cloning, so its whole payload is charged.
const inline = await matchNativeFunctions({
left: structuredClone(left),
right: structuredClone(right),
});
expect(inline.ok).toBe(false);
if (!inline.ok)
expect(inline.error._tag).toBe("AnalysisResourceConstraintError");
});
it("still charges a borrowed inventory's envelope, which matching copies into its result", async () => {
// Limitations are prefixed into the report and Evidence; 25 distinct
// 1 MiB entries are charged above the allowance although rows are few.
const record = (side: "1" | "2", limitations: readonly string[]) =>
immutableEvidence(
createEvidence(nativeMatchingTarget(side), nativeMatchingProvider, {
operation: "list_procedures",
parameters: {},
result: [ghidraRow(side, 0)],
limitations: [...limitations],
}),
);
const left = record(
"1",
Array.from({ length: 25 }, (_, i) => `${"l".repeat(1024 * 1024)}${i}`),
),
right = record("2", []);
const ledger = new Map([left, right].map((x) => [x.evidence_id, x]));
const refused = await matchNativeFunctions(
{
left: { kind: "retained-evidence", evidence_id: left.evidence_id },
right: { kind: "retained-evidence", evidence_id: right.evidence_id },
},
undefined,
(id) => ledger.get(id),
);
expect(refused.ok).toBe(false);
if (!refused.ok)
expect(refused.error._tag).toBe("AnalysisResourceConstraintError");
});
it("matches 20,000 retained Ghidra-shaped rows per side and refuses an oversized side before parsing its rows", async () => {
const large = retainedInventories((side) =>
Array.from({ length: 20_000 }, (_, i) => ghidraRow(side, i)),
);
const matched = await matchNativeFunctions(
large.references,
undefined,
large.lookup,
);
if (!matched.ok) throw matched.error;
expect(
nativeFunctionMatchingSchema.parse(matched.value.evidence.normalized_result)
.summary,
).toEqual({ matched: 20_000, only_left: 0, only_right: 0, unresolved: 0 });
// A malformed final row would be an integrity failure if parsing began.
const oversized = retainedInventories((side) => [
...Array.from({ length: 50_000 }, (_, i) => ghidraRow(side, i)),
{ address: "not an address", value: 1 },
]);
const refused = await matchNativeFunctions(
oversized.references,
undefined,
oversized.lookup,
);
expect(refused.ok).toBe(false);
if (!refused.ok)
expect(refused.error).toMatchObject({
_tag: "AnalysisResourceConstraintError",
reportedLimits: { maximum_bytes: 96 * 1024 * 1024 },
retryable: false,
});
});
it("keeps duplicate candidate storage linear and emits every input occurrence exactly once", async () => {
const entries = Array.from({ length: 1000 }, (_, i) => ({
address: `0x${i.toString(16)}`,
@@ -1,6 +1,7 @@
import { setImmediate } from "node:timers/promises";
import { AnalysisResourceConstraintError } from "../../domain/analysisErrorCore.js";
import { fixedNativeAllowance } from "../../domain/native/nativeDeadlines.js";
import type { Evidence } from "../../domain/evidence.js";
const NATIVE_MATCHING_REPRESENTATION_BYTES = 96 * 1024 * 1024;
@@ -34,6 +35,17 @@ export class NativeMatchingAdmission {
this.signal === undefined ? {} : { signal: this.signal },
);
};
/**
* Charge a borrowed record's envelope, which matching copies into its
* limitations, digests and links. Its result payloads stay resident: the
* caller charges what matching derives from them, or never parses them.
*/
async admitBorrowed(record: Evidence): Promise<void> {
await this.checkpoint(128);
for (const [key, value] of Object.entries(record))
if (key !== "raw_result" && key !== "normalized_result")
await this.admit(value);
}
async admit(root: unknown): Promise<void> {
const pending: Iterator<unknown>[] = [[root][Symbol.iterator]()];
while (pending.length) {
@@ -1,7 +1,7 @@
import type { MatchNativeFunctionsInput } from "../../contracts/native/functionMatchingContracts.js";
import { digestCanonicalValue } from "../../domain/canonicalDigest.js";
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
import type { Evidence } from "../../domain/evidence.js";
import { isImmutableEvidence, type Evidence } from "../../domain/evidence.js";
import { readEvidence } from "../../domain/evidenceValidation.js";
import { retainedEvidenceReferenceSchema } from "../../domain/evidenceReference.js";
import {
@@ -34,18 +34,30 @@ export const resolveNativeMatchingSources = async (
readonly sources: readonly Evidence[];
}> => {
const sources = new Map<string, Evidence>();
/**
* Charge only what matching copies. A retained immutable record is borrowed
* without cloning. When matching never parses its result, or charges the
* representation it derives from it (inventory rows), only its envelope is
* charged; walking the result would charge resident memory as if matching
* had allocated it.
*/
const resolve = async (
input: unknown,
path?: readonly (string | number)[],
borrowable = false,
): Promise<Evidence> => {
const reference = retainedEvidenceReferenceSchema.safeParse(input);
let raw = input;
let raw = input,
borrowed: Evidence | undefined;
if (reference.success) {
const found = resolveEvidenceInput(reference.data, lookup);
if (!found.ok) throw found.error;
raw = found.value;
if (borrowable && isImmutableEvidence(found.value))
borrowed = found.value;
}
await admission.admit(raw);
if (borrowed !== undefined) await admission.admitBorrowed(borrowed);
else await admission.admit(raw);
const checked = readEvidence(raw);
if (!checked.ok)
throw path !== undefined && !reference.success
@@ -66,8 +78,9 @@ export const resolveNativeMatchingSources = async (
raw: unknown,
path: readonly (string | number)[],
operations: readonly string[],
borrowable = false,
): Promise<Evidence> => {
const evidence = await resolve(raw, path);
const evidence = await resolve(raw, path, borrowable);
if (
evidence.predicate_type !== "rea.analysis" ||
!operations.includes(evidence.operation) ||
@@ -82,8 +95,14 @@ export const resolveNativeMatchingSources = async (
}
return evidence;
};
const left = await select(input.left, ["left"], INVENTORY_OPERATIONS);
const right = await select(input.right, ["right"], INVENTORY_OPERATIONS);
// readFunctionInventory charges each inventory row before parsing it.
const left = await select(input.left, ["left"], INVENTORY_OPERATIONS, true);
const right = await select(
input.right,
["right"],
INVENTORY_OPERATIONS,
true,
);
const observations = async (
inputs: readonly unknown[],
field: "left_observations" | "right_observations",
@@ -136,7 +155,12 @@ export const resolveNativeMatchingSources = async (
for (const source of sources.values())
for (const id of source.evidence_links)
if (!sources.has(id))
await resolve({ kind: "retained-evidence", evidence_id: id });
// Closure records are only returned for retention, never parsed.
await resolve(
{ kind: "retained-evidence", evidence_id: id },
undefined,
true,
);
return {
left,
right,
+20
View File
@@ -31,6 +31,8 @@ import {
type NativeDmgInfo,
} from "./NativeDmgOwnership.js";
const HDIUTIL_COMMAND_TIMEOUT_MS = 120_000;
const DMG_CHECKSUM_ABSENT_LIMITATION =
"hdiutil verify found no stored checksum in this disk image, so its integrity was not verified; the root SHA-256 identifies the inventoried bytes.";
const VERIFY_SETTLE_DELAYS_MS = [250, 500] as const;
// APFS ejects and busy reporting can lag a detach attempt.
const DETACH_SETTLE_DELAYS_MS = [250, 500, 1000] as const;
@@ -60,12 +62,15 @@ interface NativeDmgFailureClassification {
* localized text; the cause stays unknown unless the target cannot be opened.
*/
readonly unrecognizedImage?: boolean;
/** hdiutil found no stored checksum, so verification neither passed nor failed. */
readonly checksumAbsent?: boolean;
}
class NativeDmgCommandFailure extends ArtifactReaderFailure {
readonly processNotStarted: boolean;
readonly verifyDiagnostic: string | undefined;
readonly unrecognizedImage: boolean;
readonly checksumAbsent: boolean;
readonly exitCode: number | null;
constructor(
@@ -94,6 +99,7 @@ class NativeDmgCommandFailure extends ArtifactReaderFailure {
this.processNotStarted = options?.processNotStarted ?? false;
this.verifyDiagnostic = failure.verifyDiagnostic;
this.unrecognizedImage = failure.unrecognizedImage ?? false;
this.checksumAbsent = failure.checksumAbsent ?? false;
this.exitCode = options?.exitCode ?? null;
}
}
@@ -162,6 +168,7 @@ export class NativeDmgArtifactReader implements ArtifactReader {
#closePromise: Promise<void> | undefined;
readonly #activeDirectoryOperations = new Set<Promise<unknown>>();
#verificationProbeHandle: OwnedFileHandle | undefined;
#checksumAbsent = false;
constructor(
private readonly path: string,
@@ -184,6 +191,11 @@ export class NativeDmgArtifactReader implements ArtifactReader {
const directory = this.#directory;
if (directory === undefined)
throw new ArtifactReaderFailure("unavailable", "DMG is not attached");
if (this.#checksumAbsent)
options?.onCoverage?.({
status: "complete",
limitations: [DMG_CHECKSUM_ABSENT_LIMITATION],
});
const prefix = basename(this.path);
const entries = directory
.entries(signal, {
@@ -498,6 +510,11 @@ export class NativeDmgArtifactReader implements ArtifactReader {
} catch (cause: unknown) {
if (!(cause instanceof NativeDmgCommandFailure)) throw cause;
this.#provenance.push(command(arguments_, ["read"], cause.exitCode));
// No stored checksum (e.g. raw UDRW): attach with integrity unverified.
if (cause.checksumAbsent) {
this.#checksumAbsent = true;
return;
}
if (cause.unrecognizedImage) {
const unreadable = await this.#isUnreadableTarget(signal, cause);
this.#throwIfVerificationCancelled(signal);
@@ -852,6 +869,7 @@ const verifyFailureDiagnostic = (
// hdiutil localizes the trailing failure reason but not its own status lines.
const INVALID_CHECKSUM_LINE = /^hdiutil: verify: checksum of ".*" is INVALID$/u;
const NO_CHECKSUM_LINE = /^hdiutil: verify: ".*" has no checksum\.$/u;
const UNRECOGNIZED_IMAGE_LINE =
/^hdiutil: verify: unable to recognize ".*" as a disk image\./u;
@@ -862,6 +880,8 @@ const untranslatedVerifyFailure = (
const lines = outputLines(stdout, stderr);
if (lines.some((line) => INVALID_CHECKSUM_LINE.test(line)))
return { reason: "integrity" };
if (lines.some((line) => NO_CHECKSUM_LINE.test(line)))
return { reason: "unavailable", checksumAbsent: true };
// Malformed, unreadable and transiently busy targets all produce this line;
// only the localized reason distinguishes them, so the cause stays unknown.
if (lines.some((line) => UNRECOGNIZED_IMAGE_LINE.test(line)))
+21 -31
View File
@@ -29,6 +29,7 @@ import {
} from "./hashStableRootArtifact.js";
import type { ArtifactInventoryOptions } from "./types.js";
import { readMachoHeaderPrefixFacts } from "../../domain/apple/machoContainer.js";
import { hasDiskImageSignature } from "../../domain/apple/diskImageFormat.js";
import {
observeMzWindowsHeader,
type MzWindowsHeaderSignatureEvidence,
@@ -152,9 +153,27 @@ const classifyRootFormat = async (
evidence.prefix.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
// Fill each range: a short result is end of file, not a partial host read.
const read = async (position: number, length: number): Promise<Buffer> => {
const bytes = Buffer.alloc(length);
let filled = 0;
while (filled < length) {
abortIfNeeded(signal);
const observed = await handle.read(
bytes,
filled,
length - filled,
position + filled,
);
abortIfNeeded(signal);
if (observed.bytesRead === 0) break;
filled += observed.bytesRead;
}
return bytes.subarray(0, filled);
};
if (
lower.endsWith(".dmg") &&
(await hasKolyTrailer(handle, archiveBytes, signal))
(await hasDiskImageSignature(archiveBytes, read))
)
return "dmg";
if (await locateGodotPck(handle, archiveBytes, signal, evidence.prefix))
@@ -180,13 +199,7 @@ const classifyRootFormat = async (
if (
(format === "file" || format === "unknown") &&
!claimsExecutableHeader(evidence.prefix, archiveBytes) &&
(await locateTrailingZipArchive(archiveBytes, async (position, length) => {
abortIfNeeded(signal);
const bytes = Buffer.alloc(length);
const read = await handle.read(bytes, 0, length, position);
abortIfNeeded(signal);
return bytes.subarray(0, read.bytesRead);
})) !== undefined
(await locateTrailingZipArchive(archiveBytes, read)) !== undefined
)
return zipRootPackageFormat(lower);
if (["elf", "pe", "mach-o", "mach-o-universal"].includes(format)) {
@@ -305,26 +318,3 @@ const readRootMzWindowsHeaderSignature = async (
if (available === 0) return undefined;
return { offset, bytes: window.subarray(0, available) };
};
const hasKolyTrailer = async (
handle: FileHandle,
size: number,
signal?: AbortSignal,
): Promise<boolean> => {
if (size < 512) return false;
const trailer = Buffer.alloc(4);
let bytes = 0;
while (bytes < trailer.length) {
abortIfNeeded(signal);
const read = await handle.read(
trailer,
bytes,
trailer.length - bytes,
size - 512 + bytes,
);
abortIfNeeded(signal);
if (read.bytesRead === 0) return false;
bytes += read.bytesRead;
}
return trailer.equals(Buffer.from("koly"));
};
+3 -5
View File
@@ -12,10 +12,8 @@ import {
advertisedInputJsonSchema,
toolOutputSchemaWithMetadata,
} from "./contracts/toolSchemaMetadata.js";
import {
presentAdvertisedInputSchema,
type CompactInputSchemaPresentation,
} from "./contracts/compactInputSchema.js";
import type { CompactInputSchemaPresentation } from "./contracts/compactInputSchema.js";
import { presentAdvertisedInputProfile } from "./contracts/inputSchemaPresentation.js";
import type { McpOutputSchemaProfile } from "./config/mcpOutputSchemaProfile.js";
import {
COMPACT_INPUT_SCHEMA_BUDGET_BYTES,
@@ -62,7 +60,7 @@ const presentToolCatalog = (
? catalog
: catalog.map((tool) => ({
...tool,
input_schema: presentAdvertisedInputSchema(
input_schema: presentAdvertisedInputProfile(
tool.input_schema,
presentation,
),
+12 -2
View File
@@ -1,7 +1,11 @@
import { readArtifactInspectionSources } from "../domain/artifactInspection.js";
import type { AnalysisViewRequest } from "../domain/analysisView/analysisView.js";
import { z } from "incur";
import { parseEvidence, type Evidence } from "../domain/evidence.js";
import {
parseEvidence,
withEvidenceCallerInputPath,
type Evidence,
} from "../domain/evidence.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
@@ -180,6 +184,12 @@ export const publishCliAnalysisEvidence = async (
return projectAnalysisError(
delivered.error.retainPartialObservation(complete),
);
// This projection answers the same direct invocation as its parent. Later
// inspect_analysis_view calls own their own input and do not inherit it.
const selected =
detail === "summary" && complete.caller_input_path !== undefined
? withEvidenceCallerInputPath(delivered.value, complete.caller_input_path)
: delivered.value;
const inspectionSources =
complete.operation === "inspect_artifact"
? readArtifactInspectionSources(complete.normalized_result)
@@ -199,7 +209,7 @@ export const publishCliAnalysisEvidence = async (
).values(),
]),
complete,
delivered: delivered.value,
delivered: selected,
outputPath: absoluteCallerPath(evidenceOutput),
operation,
signal,
+1
View File
@@ -337,6 +337,7 @@ export const createCliResultOutput = (
...options,
source: value,
selected: filtered,
filterPaths,
operationFailure,
command: metadata.command,
duration: metadata.duration,
+28 -8
View File
@@ -16,6 +16,13 @@ import {
snapshotSelectionOption,
} from "./options.js";
import type { CliInstance } from "./types.js";
import { withCommandCancellation } from "./commandCancellation.js";
import {
analysisEvidenceDeliveryOptions,
cliEvidenceDeliveryInputError,
publishCliAnalysisEvidence,
} from "./analysisEvidenceDelivery.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
/** Register native load-image, memory, and annotation CLI commands. */
export const registerCoreNativeCommands = (
@@ -66,19 +73,32 @@ export const registerCoreNativeCommands = (
});
cli.command(CLI_COMMANDS.inspectNativeLoadImage, {
description:
"Verify loaded native bytes, source mappings, relocations and entry",
"Inspect native load observations and supported snapshot verification",
args: z.object({ path: z.string().describe("Local executable path") }),
options: z.object({
...analysisEvidenceDeliveryOptions,
...nativeOpenOptionFields,
}),
run: ({ args, options }) =>
logCliCommand(logger, CLI_COMMANDS.inspectNativeLoadImage, () =>
runDirectAnalysis(
args.path,
"inspect_native_load_image",
{},
directAnalysisOptionsFromCli(logger, options),
),
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.inspectNativeLoadImage, async () => {
const delivery = {
providerId: "auto",
operation: "inspect_native_load_image",
signal,
detail: options.detail,
evidenceOutput: options.evidence_output,
};
const invalid = cliEvidenceDeliveryInputError(delivery);
if (invalid !== undefined) return projectAnalysisError(invalid);
const complete = await runDirectAnalysis(
args.path,
"inspect_native_load_image",
{},
{ ...directAnalysisOptionsFromCli(logger, options), signal },
);
return publishCliAnalysisEvidence(complete, delivery);
}),
),
});
cli.command(CLI_COMMANDS.readBytes, {
+8
View File
@@ -21,6 +21,7 @@ export const finishAnalysisCommand = async <
canonicalOperation: string,
signal: AbortSignal,
fallbackResource?: string,
options: { readonly failureProjection?: "initial" | "final" } = {},
): Promise<Value | ReturnType<typeof projectAnalysisError>> => {
let result: Result<Value, AnalysisError>;
let cleanup: AnalysisError | undefined;
@@ -35,6 +36,12 @@ export const finishAnalysisCommand = async <
: new ProviderAdapterError(providerId, canonicalOperation, { cause }),
};
}
// Some one-shot commands promise the original failure snapshot even when
// close retries and mutates the producer's live cleanup status.
const initialFailure =
!result.ok && options.failureProjection === "initial"
? projectAnalysisError(result.error)
: undefined;
try {
// Resource owners may resolve their retained errors during close. This
// adapter cannot infer resolution of a producer's earlier cleanup failure.
@@ -61,6 +68,7 @@ export const finishAnalysisCommand = async <
}
}
await dispatchQueuedInterrupts();
if (initialFailure !== undefined) return initialFailure;
if (result.ok && signal.aborted)
result = {
ok: false,
+211 -63
View File
@@ -1,6 +1,8 @@
import type { Filter } from "incur";
import { err, ok, type Result } from "../domain/result.js";
import { AnalysisOutputError } from "../domain/analysisErrorCore.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
import { jsonParts } from "../domain/jsonSerialization.js";
/** Reserved control-record marker, not an advertised external JSON Schema. */
@@ -27,6 +29,21 @@ interface JsonLinesMetadataRecord {
const COLLECTION_KEYS = ["rows", "items", "records", "occurrences"] as const;
const RECORD_BOUNDARIES =
"JSON Lines needs a top-level array, or exactly one rows, items, records, or occurrences array directly in the result, under page, or under normalized_result.";
/** Objects whose collection fields JSON Lines admits, relative to the result. */
const RECORD_BOUNDARY_PATHS = [
[],
["page"],
["normalized_result"],
["normalized_result", "page"],
] as const;
/** Recovery for every JSON Lines selection refusal; the analysis result itself is valid. */
export const JSON_LINES_SELECTION_REMEDIATION =
"Use --format json for the complete document, or select one record collection with --filter-output.";
interface Collection {
readonly records: readonly unknown[];
readonly path: readonly string[];
@@ -50,42 +67,100 @@ const atPath = (value: unknown, path: readonly string[]): unknown => {
return value;
};
const collections = (
value: unknown,
prefix: readonly string[],
): Collection[] => {
if (Array.isArray(value)) return [{ records: value, path: prefix }];
if (!isRecord(value)) return [];
const found: Collection[] = [];
const collect = (
object: Record<string, unknown>,
path: readonly string[],
) => {
for (const key of COLLECTION_KEYS) {
const records = object[key];
if (Object.hasOwn(object, key) && Array.isArray(records))
found.push({ records, path: [...path, key] });
}
};
collect(value, prefix);
if (Object.hasOwn(value, "page") && isRecord(value.page))
collect(value.page, [...prefix, "page"]);
return found;
};
interface BoundaryExamination {
readonly found: readonly Collection[];
/** Collection fields that exist but are not arrays. */
readonly rejected: readonly Observation[];
}
const documentCollections = (
/** Examine exactly the documented record boundaries, once, for admission and refusal. */
const examineBoundaries = (
value: unknown,
fullOutput: boolean,
): Collection[] => {
): BoundaryExamination => {
const prefix = fullOutput ? ["data"] : [];
const data = atPath(value, prefix);
return [
...collections(data, prefix),
...collections(atPath(data, ["normalized_result"]), [
...prefix,
"normalized_result",
]),
];
const found: Collection[] = [];
const rejected: Observation[] = [];
for (const boundary of RECORD_BOUNDARY_PATHS) {
const object = atPath(data, boundary);
const path = [...prefix, ...boundary];
// A result or normalized_result may itself be the array; a page is a container.
if (Array.isArray(object) && boundary.at(-1) !== "page")
found.push({ records: object, path });
if (!isRecord(object)) continue;
for (const key of COLLECTION_KEYS) {
if (!Object.hasOwn(object, key)) continue;
const records = object[key];
if (Array.isArray(records)) found.push({ records, path: [...path, key] });
else rejected.push(observe([...path, key], records));
}
}
return { found, rejected };
};
type Shape =
| "absent"
| "null"
| "array"
| "object"
| "string"
| "number"
| "boolean";
/** One examined value that JSON Lines refused, at its emitted-document path. */
interface Observation {
readonly path: readonly (string | number)[];
readonly shape: Shape;
}
const shapeOf = (value: unknown): Shape => {
if (value === null) return "null";
if (Array.isArray(value)) return "array";
switch (typeof value) {
case "object":
return "object";
case "string":
return "string";
case "boolean":
return "boolean";
case "number":
case "bigint":
return "number";
default:
// JSON omits undefined, functions and symbols.
return "absent";
}
};
const observe = (
path: readonly (string | number)[],
value: unknown,
): Observation => ({ path, shape: shapeOf(value) });
const describe = ({ path, shape }: Observation): string =>
`${path.length === 0 ? "the result" : path.join(".")} is ${
shape === "absent" || shape === "null"
? shape
: shape === "array" || shape === "object"
? `an ${shape}`
: `a ${shape}`
}`;
/** Caller-visible key paths, before any slice, selected by --filter-output. */
const selectionPaths = (
paths: ReturnType<typeof Filter.parse> | undefined,
): string[][] => {
const selected = new Map<string, string[]>();
for (const path of paths ?? []) {
const keys: string[] = [];
for (const segment of path) {
if (!("key" in segment)) break;
keys.push(segment.key);
}
selected.set(JSON.stringify(keys), keys);
}
return [...selected.values()];
};
/** Shallow copies only ancestors of the emitted array; every other fact survives. */
@@ -123,40 +198,116 @@ const withoutRecords = (
return context;
};
/** Project a clear collection without interpreting or whitelisting its context. */
export const prepareJsonLines = (
interface JsonLinesOptions {
readonly command: string;
readonly source?: unknown;
readonly filter?: string;
/** Parsed --filter-output paths, used only to name the refused selection. */
readonly filterPaths?: ReturnType<typeof Filter.parse>;
readonly fullOutput?: boolean;
}
/**
* Choose the one clear collection, or refuse the caller's selection. The
* structured `observed` facts are the contract; the message derives from them.
*/
const selectCollection = (
selected: unknown,
options: {
readonly command: string;
readonly source?: unknown;
readonly filter?: string;
readonly fullOutput?: boolean;
},
): Result<JsonLinesPlan, AnalysisOutputError> => {
const found = documentCollections(selected, options.fullOutput === true);
const reject = (reason: string) =>
err(
new AnalysisOutputError(options.command, reason, {
remediationAction:
"Use --format json for the complete document, or select one record collection with --filter-output.",
}),
source: unknown,
options: JsonLinesOptions,
{ found, rejected }: BoundaryExamination,
): Result<Collection, AnalysisInputError> => {
const prefix = options.fullOutput === true ? ["data"] : [];
const filtered = options.filter !== undefined;
const selections = selectionPaths(options.filterPaths).filter(
(path) => path.length > 0,
);
const subject =
filtered && selections.length > 0
? `The --filter-output selection ${selections.map((path) => path.join(".")).join(", ")}`
: "The command result";
const refuse = (
statement: string,
observed: readonly Observation[],
rule: string,
) => {
const observation = `${subject} ${statement}: ${observed.map(describe).join("; ")}.`;
return err(
new AnalysisInputError(
options.command,
{
userMessage: `${observation} ${rule}`,
remediationAction: JSON_LINES_SELECTION_REMEDIATION,
},
[
{
path: [filtered ? "filter-output" : "format"],
reason: "invalid_value",
message: observation,
observed: observed.map(({ path, shape }) => ({
path: [...path],
shape,
})),
},
],
),
);
if (found.length !== 1)
return reject(
found.length === 0
? "JSON Lines requires an array or one explicit rows, items, records, or occurrences collection."
: "JSON Lines cannot choose between multiple record collections. Select one collection explicitly.",
};
if (found.length > 1)
return refuse(
"is not one JSON Lines record collection",
found.map(({ path }) => observe(path, [])),
"Select one with --filter-output.",
);
const collection = found[0];
if (collection === undefined)
return reject("The record collection is unavailable.");
for (const record of collection.records) {
const [collection] = found;
if (collection === undefined) {
const [selection] = selections;
// A non-array collection field is the most specific fact. Otherwise one
// named selection is described by its source value, since a filter that
// matches nothing leaves no selected value to describe.
const observed =
rejected.length > 0
? rejected
: [
selections.length === 1 && selection !== undefined
? observe(
[...prefix, ...selection],
atPath(atPath(source, prefix), selection),
)
: observe(prefix, atPath(selected, prefix)),
];
return refuse(
"is not a JSON Lines record collection",
observed,
RECORD_BOUNDARIES,
);
}
for (const [index, record] of collection.records.entries()) {
if (isRecord(record) && Object.hasOwn(record, "$rea"))
return reject(
"A data record uses the reserved $rea JSON Lines control marker.",
return refuse(
"uses the field JSON Lines reserves for control records",
[observe([...collection.path, index, "$rea"], record.$rea)],
"Nested $rea values are ordinary data.",
);
}
return ok(collection);
};
/**
* Project a clear collection without interpreting or whitelisting its context.
* A refusal is the caller's output selection, not a failed analysis: the same
* result remains available through --format json.
*/
export const prepareJsonLines = (
selected: unknown,
options: JsonLinesOptions,
): Result<JsonLinesPlan, AnalysisInputError> => {
const fullOutput = options.fullOutput === true;
const examination = examineBoundaries(selected, fullOutput);
const source = Object.hasOwn(options, "source") ? options.source : selected;
const chosen = selectCollection(selected, source, options, examination);
if (!chosen.ok) return chosen;
const collection = chosen.value;
const originalRecords = atPath(source, collection.path);
const sourceMatches = Array.isArray(originalRecords);
let context = withoutRecords(
@@ -168,10 +319,7 @@ export const prepareJsonLines = (
if (options.filter !== undefined && sourceMatches) {
// Excluded peer rows are data, not context. A filter can disambiguate them
// without restoring an unselected collection into the final record.
for (const peer of documentCollections(
source,
options.fullOutput === true,
)) {
for (const peer of examineBoundaries(source, fullOutput).found) {
if (peer.path.join(".") !== collection.path.join("."))
context = withoutRecords(context, selected, peer.path, true);
}
+5
View File
@@ -1,5 +1,6 @@
import { constants } from "node:buffer";
import type { Writable } from "node:stream";
import type { Filter } from "incur";
import { estimateTokenCount } from "tokenx";
import { AnalysisResourceConstraintError } from "../domain/analysisErrorCore.js";
@@ -29,6 +30,7 @@ interface CliJsonLinesInput {
readonly fullOutput: boolean;
readonly tokenCount: boolean;
readonly filterOutput: string | undefined;
readonly filterPaths: ReturnType<typeof Filter.parse> | undefined;
}
/** Render one already classified result, sharing a plan for output and counting. */
@@ -57,6 +59,9 @@ export const writeCliJsonLines = async (
...(options.filterOutput === undefined
? {}
: { filter: options.filterOutput }),
...(options.filterPaths === undefined
? {}
: { filterPaths: options.filterPaths }),
fullOutput: options.fullOutput,
})
: undefined;
+36 -27
View File
@@ -2,15 +2,17 @@ import { Cli } from "incur";
import { z } from "zod";
import { captureBrowserScenario } from "./application/BrowserScenarioCaptureService.js";
import { finishAnalysisCommand } from "./cli/finishAnalysisCommand.js";
import { withCommandCancellation } from "./cli/commandCancellation.js";
import {
analysisEvidenceDeliveryOptions,
cliEvidenceDeliveryInputError,
publishCliAnalysisEvidence,
} from "./cli/analysisEvidenceDelivery.js";
import { CLI_COMMANDS } from "./cliCommandNames.js";
import { parseCliJsonInput, resolveCliJsonPaths } from "./cliJsonInput.js";
import { logCliCommand } from "./cliLogging.js";
import { projectAnalysisError } from "./domain/analysisErrorProjection.js";
import {
BrowserObservationError,
browserCleanupObservation,
} from "./domain/browserObservationError.js";
import { analysisInputErrorFromIssues } from "./domain/inputIssueProjection.js";
import { browserScenarioSchema } from "./domain/browserScenario.js";
import type { BrowserScenarioCapturePort } from "./application/BrowserScenarioCapturePort.js";
@@ -38,9 +40,16 @@ export const registerBrowserScenarioCommands = (
.string()
.describe("Inline browser scenario JSON or JSON file path"),
}),
run: ({ args }) =>
options: z.object(analysisEvidenceDeliveryOptions),
run: ({ args, options: delivery }) =>
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.captureBrowserScenario, async () => {
const deliveryError = cliEvidenceDeliveryInputError({
detail: delivery.detail,
evidenceOutput: delivery.evidence_output,
operation: OPERATION,
});
if (deliveryError !== undefined) return cliError(deliveryError);
const input = await parseCliJsonInput(
args.inputJson,
OPERATION,
@@ -67,28 +76,28 @@ export const registerBrowserScenarioCommands = (
createBrowserScenarioProvider(environment),
)
: options.createProvider(environment);
let result:
| Awaited<ReturnType<typeof captureBrowserScenario>>
| undefined;
try {
result = await captureBrowserScenario(provider, scenario.data, {
signal,
});
return result.ok ? result.value : cliError(result.error);
} finally {
try {
await provider?.close?.();
} catch (cause: unknown) {
// Failed captures already retain their primary error and partial
// observations. A cleanup retry must not replace that outcome.
if (result?.ok)
throw new BrowserObservationError(OPERATION, "cleanup_failed", {
cause,
cleanup: browserCleanupObservation(cause),
partialObservation: result.value,
});
}
}
const result = await finishAnalysisCommand(
() => captureBrowserScenario(provider, scenario.data, { signal }),
{
close: async () => {
await provider?.close?.();
},
},
"rea-playwright-browser-scenario",
OPERATION,
signal,
undefined,
{ failureProjection: "initial" },
);
if (!("evidence_id" in result))
return { error: "Browser scenario capture failed", ...result };
return publishCliAnalysisEvidence(result, {
providerId: "rea-playwright-browser-scenario",
operation: OPERATION,
signal,
detail: delivery.detail,
evidenceOutput: delivery.evidence_output,
});
}),
),
});
+1 -1
View File
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views rLine truncated
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where viewLine truncated
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
+13 -4
View File
@@ -1,6 +1,10 @@
import type { ToolContract } from "./toolContractTypes.js";
import { toolContractMetadata } from "./toolEffects.js";
import { evidenceResultOf } from "./toolOutputSchemaPrimitives.js";
import {
analysisDetailOutputSchemaOf,
analysisDetailSchema,
} from "./analysisDetail.js";
import { browserScenarioSummaryViewResultSchema } from "../domain/analysisView/analysisView.js";
import { browserScenarioInputSchema } from "../domain/browserScenario.js";
import { browserScenarioCaptureSchema } from "../domain/browserScenarioCapture.js";
import type { JsonValue } from "../domain/jsonValue.js";
@@ -26,10 +30,15 @@ export const BROWSER_SCENARIO_TOOL_CONTRACTS = [
name: "capture_browser_scenario",
...toolContractMetadata("capture_browser_scenario"),
description:
"Run a controlled browser scenario when passive observation cannot exercise the application. Launch the selected executable or connect to a loopback CDP target, provide a start URL and actions, and select storage or artifacts when needed. A request action sends one HTTP request to the start origin through the page's own fetch, with header values from secrets, page cookies or local storage; write methods change application state. Environment, empty storage, a URL-only final capture, and measured event-retention budgets are defaults. capture.network selects request_body, response_body, and header_values independently; capture.limits lets callers raise or lower the event count, retained-allocation estimates, network transaction count, and network content budgets. Selected content is returned inline with transaction and event references, byte digests, and explicit unavailable states. Budget omissions identify affected event kinds and network transactions. Credentials and declared secrets are redacted. Agents can run their own local fixture servers to control application responses. Actions run until completion or cancellation unless timeout_ms is set. Results are returned inline.",
"Run a controlled browser scenario when passive observation cannot exercise the application. Launch the selected executable or connect to a loopback CDP target, provide a start URL and actions, and select storage or artifacts when needed. A request action sends one HTTP request to the start origin through the page's own fetch, with header values from secrets, page cookies or local storage; write methods change application state. Environment, empty storage, a URL-only final capture, and measured event-retention budgets are defaults. capture.network selects request_body, response_body, and header_values independently; capture.limits lets callers raise or lower the event count, retained-allocation estimates, network transaction count, and network content budgets. Selected content is returned inline with transaction and event references, byte digests, and explicit unavailable states. Budget omissions identify affected event kinds and network transactions. Credentials and declared secrets are redacted. Agents can run their own local fixture servers to control application responses. Actions run until completion or cancellation unless timeout_ms is set. Default detail complete returns the capture inline. Detail summary retains complete Evidence and returns step outcomes, the final or first unsuccessful step’s accessibility tree, and event/network counts. Read exact steps with selected artifacts, UTF-16 text pages, transactions with opt-in content, or events filtered by step, kind and transaction, through inspect_analysis_view kind browser-scenario.",
kind: "browser-provider",
inputSchema: browserScenarioInputSchema,
outputSchema: evidenceResultOf(browserScenarioCaptureSchema),
inputSchema: browserScenarioInputSchema.extend({
detail: analysisDetailSchema,
}),
outputSchema: analysisDetailOutputSchemaOf(
browserScenarioCaptureSchema,
browserScenarioSummaryViewResultSchema,
),
examples: [
{ title: "Capture a deterministic browser scenario", input: example },
],
+186 -12
View File
@@ -410,24 +410,27 @@ describe("compact input schema budget and reduction", () => {
expect(presentation).toBe("reduced");
expect(schema.properties).toEqual({
capture: { type: "object", description: "Complete capture." },
observations: {
type: "array",
items: {
type: ["null", "boolean", "object", "array", "number", "string"],
items: {},
},
},
observations: { type: "array", items: { type: "object" } },
label: { type: ["string", "null"] },
});
const validate = new Ajv2020({ strict: false }).compile(schema);
expect(validate({ capture: {}, observations: [], label: null })).toBe(true);
const observation = {
captured_url: "https://example.test/",
captured_at: "2026-10-25T00:00:00Z",
status_code: 200,
};
const canonicalObservations = new Ajv2020({ strict: false }).compile(
original.$defs.observations,
);
expect(canonicalObservations([observation])).toBe(true);
expect(canonicalObservations(["text"])).toBe(false);
expect(
validate({
capture: {},
observations: [[[null, true, 42, "nested"], { values: [[], [false]] }]],
label: null,
}),
validate({ capture: {}, observations: [observation], label: null }),
).toBe(true);
expect(validate({ capture: {}, observations: ["text"], label: null })).toBe(
false,
);
expect(validate({ capture: {}, observations: [], label: "capture" })).toBe(
true,
);
@@ -697,3 +700,174 @@ it("includes restored root identity in the compact byte budget", () => {
expect(serializedBytes(result.schema)).toBeLessThanOrEqual(512);
expect(result.schema.properties).toEqual({ value: { type: "number" } });
});
// An untyped primitive enum is gateway-incompatible, so these schemas take
// the reduced projection independently of the budget.
const reducedPresentation = (
properties: Record<string, unknown>,
$defs: Record<string, unknown> = {},
) => {
const canonical = {
type: "object",
$defs,
properties: { mode: { enum: ["fast", "safe"] }, ...properties },
};
const result = compactAdvertisedInputSchema(canonical, 13 * 1024);
expect(result.presentation).toBe("reduced");
const { mode: _mode, ...advertised } = result.schema.properties as Record<
string,
unknown
>;
return {
advertised,
validateCanonical: new Ajv2020({ strict: false }).compile(canonical),
validateAdvertised: new Ajv2020({ strict: false }).compile(result.schema),
};
};
const reducedArrayCases = [
[
"string elements",
{ type: "array", items: { type: "string", minLength: 1 } },
{ type: "array", items: { type: "string" } },
[["a", "b"]],
],
[
"object union elements",
{
type: "array",
minItems: 1,
items: {
oneOf: [
{
type: "object",
properties: { kind: { const: "click" } },
required: ["kind"],
},
{
type: "object",
properties: { kind: { const: "type" } },
required: ["kind"],
},
],
},
},
{ type: "array", items: { type: "object" } },
[[{ kind: "click" }, { kind: "type" }]],
],
[
"referenced elements",
{ type: "array", items: { $ref: "#/$defs/ordinal" } },
{ type: "array", items: { type: "integer" } },
[[0, 7]],
],
[
"nested array elements",
{
type: "array",
items: { type: "array", items: { type: "number" } },
},
{ type: "array", items: { type: "array", items: { type: "number" } } },
[[[1.5], []]],
],
[
"mixed scalar and object elements",
{
type: "array",
items: { anyOf: [{ type: "string" }, { type: "object" }] },
},
{ type: "array", items: { type: ["string", "object"] } },
[["text", {}]],
],
[
"elements that may be arrays or scalars",
{
type: "array",
items: {
anyOf: [
{ type: "string" },
{ type: "array", items: { type: "string" } },
],
},
},
{ type: "array", items: {} },
[["text", ["nested"]]],
],
[
"tuple elements",
{
type: "array",
prefixItems: [{ type: "integer" }],
items: { type: "string" },
},
{ type: "array", items: {} },
[[1, "rest"]],
],
[
"nullable arrays",
{
anyOf: [{ type: "array", items: { type: "string" } }, { type: "null" }],
},
{
anyOf: [{ type: "array", items: { type: "string" } }, { type: "null" }],
},
[null, ["text"]],
],
] as const;
describe("reduced array item presentation", () => {
it.each(reducedArrayCases)(
"advertises %s without placing items beside a non-array type",
(_label, property, expected, canonicalValues) => {
const { advertised, validateCanonical, validateAdvertised } =
reducedPresentation(
{ values: property },
{ ordinal: { type: "integer", minimum: 0 } },
);
expect(advertised).toEqual({ values: expected });
for (const values of canonicalValues) {
expect(validateCanonical({ values })).toBe(true);
expect(validateAdvertised({ values })).toBe(true);
}
},
);
it("stops deriving element types at the nested array depth bound", () => {
let property: Record<string, unknown> = { type: "string" };
for (let level = 0; level < 5; level += 1)
property = { type: "array", items: property };
const { advertised, validateAdvertised } = reducedPresentation({
values: property,
});
expect(advertised).toEqual({
values: {
type: "array",
items: {
type: "array",
items: { type: "array", items: { type: "array", items: {} } },
},
},
});
expect(validateAdvertised({ values: [[[[["deep"]]]]] })).toBe(true);
});
it("keeps an inline forbidden value as the empty schema under not", () => {
const canonical = {
type: "object",
properties: { view: { type: "string" } },
if: { properties: { view: { const: "x" } }, required: ["view"] },
then: { required: ["view"] },
else: { properties: { view: { not: {} } } },
};
const { schema, presentation } = compactAdvertisedInputSchema(
canonical,
13 * 1024,
);
expect(presentation).toBe("inline");
expect(schema).toHaveProperty(["else", "properties", "view", "not"], {});
const canonicalValidate = new Ajv2020({ strict: false }).compile(canonical);
const advertisedValidate = new Ajv2020({ strict: false }).compile(schema);
for (const input of [{}, { view: "x" }, { view: "y" }])
expect(advertisedValidate(input)).toBe(canonicalValidate(input));
});
});
+143 -9
View File
@@ -28,6 +28,8 @@ const jsonValueTypes = [
];
// An empty item schema accepts every JSON value, including arbitrarily nested
// arrays, while satisfying clients that require `items` for array types.
// Gemini rejects `items` beside a type list; the compact catalog test fails
// if a tool would advertise this presentation.
const jsonValueSchema = () => ({ type: [...jsonValueTypes], items: {} });
const enumValueTypes = (values: readonly unknown[]): readonly string[] => [
@@ -240,6 +242,18 @@ const inlineWithinBudget = (
if (!first) yield ",";
first = false;
yield JSON.stringify(key) + ":";
// `not: {}` forbids a value; no client infers a type for a value it
// must omit, and the any-value list would place `items` beside a
// non-array type, which Gemini function declarations reject.
if (
position === "schema" &&
key === "not" &&
isSchemaObject(child) &&
Object.keys(child).length === 0
) {
yield "{}";
continue;
}
const childPosition =
position === "map"
? "schema"
@@ -323,6 +337,106 @@ const detectedTypes = (
return [...new Set(types.flatMap((type) => type ?? []))];
};
interface ScopedSchema {
readonly schema: Record<string, unknown>;
readonly scope: JsonSchemaScope;
}
/**
* Collect the item schema of every array alternative. Tuples, boolean item
* schemas and unresolved references are unknown because one item type would
* not describe every accepted element.
*/
const arrayItemAlternatives = (
schema: unknown,
references: JsonSchemaReferences,
inherited: JsonSchemaScope | undefined,
active: ReadonlySet<object> = new Set(),
): readonly ScopedSchema[] | undefined => {
if (!isSchemaObject(schema)) return undefined;
const scope = references.enter(schema, inherited);
if (scope === undefined) return undefined;
if (typeof schema.$ref === "string") {
if (active.has(schema)) return undefined;
const target = references.resolve(scope, schema.$ref);
return target.kind === "target"
? arrayItemAlternatives(
target.value,
references,
scope,
new Set([...active, schema]),
)
: undefined;
}
const types =
typeof schema.type === "string"
? [schema.type]
: Array.isArray(schema.type)
? schema.type
: undefined;
if (types !== undefined) {
if (!types.includes("array")) return [];
return isSchemaObject(schema.items) && schema.prefixItems === undefined
? [{ schema: schema.items, scope }]
: undefined;
}
const alternatives = Array.isArray(schema.anyOf)
? schema.anyOf
: Array.isArray(schema.oneOf)
? schema.oneOf
: [];
if (alternatives.length === 0) return undefined;
const nested = alternatives.map((branch) =>
arrayItemAlternatives(branch, references, scope, active),
);
return nested.some((items) => items === undefined)
? undefined
: nested.flatMap((items) => items ?? []);
};
/** Nested arrays beyond this level advertise an unconstrained item schema. */
const REDUCED_ITEM_MAX_DEPTH = 3;
/**
* Advertise the element types the canonical item schemas accept. Gemini
* function declarations permit `items` only beside the single `array` type,
* so an element that may be an array alongside other types, or whose type is
* unknown, uses the empty schema: it accepts every element and still gives
* clients that require `items` an item schema.
*/
const reducedItemSchema = (
alternatives: readonly ScopedSchema[] | undefined,
references: JsonSchemaReferences,
depth = 1,
): Record<string, unknown> => {
if (
alternatives === undefined ||
alternatives.length === 0 ||
depth > REDUCED_ITEM_MAX_DEPTH
)
return {};
const types: string[] = [];
for (const { schema, scope } of alternatives) {
const detected = detectedTypes(schema, references, scope);
if (detected === undefined || detected.length === 0) return {};
types.push(...detected);
}
const unique = [...new Set(types)];
if (!unique.includes("array"))
return { type: unique.length === 1 ? unique[0] : unique };
if (unique.length > 1) return {};
const nested: ScopedSchema[] = [];
for (const { schema, scope } of alternatives) {
const items = arrayItemAlternatives(schema, references, scope);
if (items === undefined) return { type: "array", items: {} };
nested.push(...items);
}
return {
type: "array",
items: reducedItemSchema(nested, references, depth + 1),
};
};
/** Give homogeneous array-item unions the type required by strict gateways. */
const typeArrayItemUnions = (
schema: Record<string, unknown>,
@@ -406,6 +520,34 @@ const schemaChildrenAndMaps = (
return [];
});
/**
* Present a reduced property's reference-resolved types. Reduced inputs omit
* nested validation, but clients still require an item schema for an array
* declaration, and Gemini accepts `items` only beside the single `array` type,
* so a property that may be an array or another type lists typed branches.
*/
const reducedPropertyTypes = (
value: unknown,
references: JsonSchemaReferences,
): Record<string, unknown> => {
const types = detectedTypes(value, references, references.rootScope);
if (types === undefined || types.length === 0) return {};
if (!types.includes("array"))
return { type: types.length === 1 ? types[0] : types };
const array = {
type: "array",
items: reducedItemSchema(
arrayItemAlternatives(value, references, references.rootScope),
references,
),
};
return types.length === 1
? array
: {
anyOf: types.map((type) => (type === "array" ? array : { type })),
};
};
const REDUCED_ANNOTATION =
"Advertised in reduced form by the compact input schema profile; the server still validates complete canonical inputs.";
@@ -441,21 +583,13 @@ const reduceToShallowProperties = (
const references = new JsonSchemaReferences(schema);
const shallow = Object.fromEntries(
Object.entries(properties).map(([name, value]) => {
const types = detectedTypes(value, references, references.rootScope);
const type = types?.length === 1 ? types[0] : types;
const description =
isSchemaObject(value) && typeof value.description === "string"
? { description: value.description }
: {};
return [
name,
{
...(type === undefined ? {} : { type }),
// Reduced inputs intentionally omit nested validation, but clients
// still require an item schema to accept an array declaration.
...(types?.includes("array") ? { items: jsonValueSchema() } : {}),
...description,
},
{ ...reducedPropertyTypes(value, references), ...description },
];
}),
);
@@ -0,0 +1,264 @@
import { Ajv } from "ajv";
import { Ajv2020 } from "ajv/dist/2020.js";
import { expect, it } from "vitest";
import { advertisedInputExamples } from "./advertisedInputExamples.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
fitAdvertisedInputSchema,
presentInputJsonSchema,
summarizedInputSchemaCandidate,
summarizedSchema,
} from "./inputSchemaPresentation.js";
import {
schemaArrays,
schemaChildren,
schemaLiteralValues,
schemaMaps,
} from "./jsonSchemaKeywords.js";
import { TOOL_CONTRACTS } from "./toolContracts.js";
type Schema = Record<string, unknown>;
const bytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
const isObject = (value: unknown): value is Schema =>
typeof value === "object" && value !== null && !Array.isArray(value);
/** The self-contained projection before fitting, with its advertised examples. */
const completeProjection = (
contract: (typeof TOOL_CONTRACTS)[number],
): Schema => {
const inline = contract.inputSchema["~standard"].jsonSchema.input({
target: "draft-2020-12",
libraryOptions: { reused: "inline" },
}) as Schema;
const examples = advertisedInputExamples(contract);
return {
...presentInputJsonSchema(inline, () => undefined),
...(examples.length === 0 ? {} : { examples }),
};
};
/** Every subschema in a schema position, ignoring literal data. */
const subschemas = function* (schema: unknown): Generator<Schema> {
if (!isObject(schema)) return;
yield schema;
for (const [key, child] of Object.entries(schema)) {
if (schemaLiteralValues.has(key)) continue;
if (schemaMaps.has(key) && isObject(child))
for (const nested of Object.values(child)) yield* subschemas(nested);
else if (schemaArrays.has(key) && Array.isArray(child))
for (const nested of child) yield* subschemas(nested);
else if (schemaChildren.has(key))
for (const nested of Array.isArray(child) ? child : [child])
yield* subschemas(nested);
}
};
/** Every JSON value nested in a value, including the value itself. */
const values = function* (value: unknown): Generator<unknown> {
yield value;
if (Array.isArray(value)) for (const item of value) yield* values(item);
else if (isObject(value))
for (const item of Object.values(value)) yield* values(item);
};
const GENERIC_PROBES: readonly unknown[] = [
null,
true,
false,
0,
1,
-1,
1.5,
"",
"a",
[],
[null],
[1],
["a"],
["a", 1],
[[]],
[{}],
{},
{ kind: "a" },
];
/** Assert a summary accepts every probe the complete schema accepts. */
const expectSummaryWidens = (
schemas: Iterable<Schema>,
probes: readonly unknown[],
dialect: "draft-2020-12" | "draft-07" = "draft-2020-12",
): number => {
const options = { strict: false, validateFormats: false };
const ajv = dialect === "draft-07" ? new Ajv(options) : new Ajv2020(options);
const seen = new Set<string>();
const failures: string[] = [];
for (const schema of schemas) {
const key = JSON.stringify(schema);
if (seen.has(key)) continue;
seen.add(key);
const summary = summarizedSchema(schema);
const complete = ajv.compile(schema);
const summarized = ajv.compile(summary);
for (const probe of probes)
if (complete(probe) && !summarized(probe))
failures.push(
`${key.slice(0, 200)} -> ${JSON.stringify(summary)} rejects ${JSON.stringify(probe).slice(0, 200)}`,
);
ajv.removeSchema();
}
expect(failures).toEqual([]);
return seen.size;
};
const SYNTHETIC_SUMMARIES: readonly Schema[] = [
// 2020-12 tuple positions with a differently typed rest.
{
type: "array",
prefixItems: [{ type: "string" }],
items: { type: "number" },
},
{ type: "array", prefixItems: [{ type: "string" }] },
{ type: "array", prefixItems: [{ type: "string" }], items: false },
{ type: "array", items: { type: "string" }, contains: { const: "a" } },
{
anyOf: [
{ type: "array", prefixItems: [{ type: "boolean" }] },
{ type: "null" },
],
},
{ type: ["array", "null"], items: { type: "string" }, minItems: 1 },
{ type: "object", properties: { a: { type: "string" } }, required: ["a"] },
{ not: { type: "string" } },
{ allOf: [{ type: "string" }, { minLength: 1 }] },
{ oneOf: [{ const: 1 }, { enum: ["a", null] }] },
{
if: { type: "string" },
then: { minLength: 1 },
else: { type: "number" },
},
];
const DRAFT_07_TUPLES: readonly Schema[] = [
{
type: "array",
items: [{ type: "string" }],
additionalItems: { type: "number" },
},
{ type: "array", items: [{ type: "string" }] },
{ type: "array", items: [{ type: "string" }], additionalItems: false },
];
it("summarizes synthetic schemas without rejecting values they accept", () => {
expect(
summarizedSchema({
type: "array",
prefixItems: [{ type: "string" }],
items: { type: "number" },
}),
).toEqual({ type: "array", items: { type: ["string", "number"] } });
expectSummaryWidens(SYNTHETIC_SUMMARIES, [
...GENERIC_PROBES,
[true],
["a", "b", 2],
[1, "a"],
[false, null],
]);
expectSummaryWidens(
DRAFT_07_TUPLES,
[...GENERIC_PROBES, ["a", 2], ["a", "b"]],
"draft-07",
);
});
it("summarizes every catalog input subschema without rejecting values it accepts", () => {
const projections = TOOL_CONTRACTS.map(completeProjection);
// Canonical example values reach deep nested subschemas with valid data.
const probes = [
...GENERIC_PROBES,
...new Map(
projections
.flatMap((schema) =>
Array.isArray(schema.examples) ? schema.examples : [],
)
.flatMap((example) => [...values(example)])
.map((value) => [JSON.stringify(value), value] as const),
).values(),
];
const checked = expectSummaryWidens(
projections.flatMap((schema) =>
[...subschemas(schema)].filter((node) => node !== schema),
),
probes,
);
expect(checked).toBeGreaterThan(1000);
}, 120_000);
/** Assert no candidate keeping more levels than the fitted one also fits. */
const expectDeepestFittingCandidate = (schema: Schema): void => {
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const described = { ...schema, description: fitted.description };
const candidates = Array.from({ length: 40 }, (_, index) =>
summarizedInputSchemaCandidate(described, index + 1),
);
const deepest = candidates.findLastIndex(
(candidate) => bytes(candidate) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(deepest).toBeGreaterThanOrEqual(0);
expect(fitted).toEqual(candidates[deepest]);
};
it("keeps a deeper fitting candidate when a shallower summary is longer", () => {
// A nullable array summary needs anyOf, so it is longer than the schema.
const nullable = { type: ["array", "null"], items: { type: "string" } };
const schema = {
type: "object",
properties: {
...Object.fromEntries(
Array.from({ length: 1_000 }, (_, index) => [
`list_${index}`,
nullable,
]),
),
// Retaining its third level would overflow; its second level fits.
large: {
type: "object",
properties: {
inner: {
type: "object",
properties: {
text: { type: "string", description: "x".repeat(30_000) },
},
},
},
},
},
};
// Summarizing every list at the first level would overflow by itself.
expect(1_000 * bytes(summarizedSchema(nullable))).toBeGreaterThan(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const properties = fitted.properties as Record<string, any>;
expect(properties.list_0).toEqual(nullable);
expect(properties.large.properties.inner).toEqual({ type: "object" });
expectDeepestFittingCandidate(schema);
});
it("fits oversized catalog input schemas at their deepest fitting level", () => {
const oversized = TOOL_CONTRACTS.map(completeProjection).filter(
(schema) => bytes(schema) > ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(oversized.length).toBeGreaterThan(0);
for (const schema of oversized) expectDeepestFittingCandidate(schema);
}, 120_000);
@@ -0,0 +1,371 @@
import { Ajv2020 } from "ajv/dist/2020.js";
import { describe, expect, it } from "vitest";
import { presentAdvertisedInputSchema } from "./compactInputSchema.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
fitAdvertisedInputSchema,
presentAdvertisedInputProfile,
presentInputJsonSchema,
} from "./inputSchemaPresentation.js";
const bytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
const ajv = () => new Ajv2020({ strict: false, validateFormats: false });
/** A strict object `levels` deep with `width` distinct padded string leaves. */
const record = (
levels: number,
width: number,
pad = 80,
path = "f",
): Record<string, unknown> => ({
type: "object",
properties: Object.fromEntries(
Array.from({ length: width }, (_, index) => [
`field_${index}`,
levels <= 1
? {
type: "string",
minLength: 1,
description: `Leaf ${path}.${index} ${"x".repeat(pad)}`,
}
: {
...record(levels - 1, width, pad, `${path}.${index}`),
description: `Level ${index}`,
},
]),
),
required: ["field_0"],
additionalProperties: false,
});
const instance = (levels: number, width: number): Record<string, any> =>
Object.fromEntries(
Array.from({ length: width }, (_, index) => [
`field_${index}`,
levels <= 1 ? "value" : instance(levels - 1, width),
]),
);
const expectWidened = (
original: Record<string, unknown>,
fitted: Record<string, unknown>,
inputs: readonly unknown[],
) => {
const validateOriginal = ajv().compile(original);
const validateFitted = ajv().compile(fitted);
for (const input of inputs)
if (validateOriginal(input))
expect(validateFitted(input), JSON.stringify(validateFitted.errors)).toBe(
true,
);
};
it("returns schemas that already fit unchanged", () => {
const schema = {
type: "object",
properties: { capture: record(3, 4) },
examples: [{ capture: instance(3, 4) }],
};
expect(bytes(schema)).toBeLessThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
expect(fitAdvertisedInputSchema(schema)).toBe(schema);
});
it("points repeated sibling properties at the first instead of repeating them", () => {
const capture = record(3, 7, 40);
const schema = {
type: "object",
description: "Compare two captures.",
properties: {
before: { ...capture, description: "Earlier capture." },
after: { ...capture, description: "Later capture." },
unrelated: { type: "string" },
},
required: ["before", "after"],
additionalProperties: false,
examples: [{ before: instance(3, 7), after: instance(3, 7) }],
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
expect(bytes(capture)).toBeLessThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(fitted).toMatchObject({
description: expect.stringMatching(
/^Compare two captures\. .*summarized form.*complete canonical inputs\.$/u,
),
properties: {
before: schema.properties.before,
after: {
type: "object",
description: "Later capture. Same schema as `before`.",
},
unrelated: { type: "string" },
},
required: ["before", "after"],
additionalProperties: false,
examples: schema.examples,
});
expect(Object.keys(fitted.properties as object)).toEqual([
"before",
"after",
"unrelated",
]);
expect(ajv().validateSchema(fitted)).toBe(true);
expectWidened(schema, fitted, [
...schema.examples,
{ before: instance(3, 7), after: { field_0: { field_0: {} } } },
]);
});
it("keeps the deepest instance level that fits and summarizes below it", () => {
const schema = {
type: "object",
properties: { capture: record(4, 6) },
additionalProperties: false,
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const capture = (fitted.properties as Record<string, any>).capture;
// Third-level objects keep their properties; fourth-level ones are summarized.
const third = capture.properties.field_1.properties.field_2;
expect(third).toMatchObject({
required: ["field_0"],
additionalProperties: false,
});
expect(third.properties.field_3).toEqual({
type: "object",
description: "Level 3",
});
expectWidened(schema, fitted, [
{ capture: instance(4, 6) },
{ capture: { field_0: { field_0: { field_0: { field_0: "" } } } } },
]);
});
it("counts combinator and conditional branches at their instance level", () => {
const branch = (kind: string) => ({
type: "object",
properties: {
kind: { type: "string", enum: [kind] },
body: record(3, 9),
},
required: ["kind", "body"],
additionalProperties: false,
});
const schema = {
type: "object",
properties: {
value: { anyOf: [branch("a"), branch("b"), { type: "null" }] },
},
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
const value = (fitted.properties as Record<string, any>).value;
// The union keeps every branch and its own properties at level two.
expect(value.anyOf).toHaveLength(3);
expect(value.anyOf[0].properties.kind).toEqual({
type: "string",
enum: ["a"],
});
expect(value.anyOf[2]).toEqual({ type: "null" });
});
it("relaxes nonmonotone positions instead of narrowing them", () => {
const deep = record(3, 7);
const schema = {
type: "object",
properties: {
exactly_one: {
oneOf: [
{ ...deep, required: ["field_0"] },
{ ...deep, required: ["field_1"] },
],
},
excluded: { type: "object", not: deep },
conditional: {
type: "object",
if: deep,
then: { required: ["field_0"] },
else: { required: ["field_1"] },
},
bounded: { type: "array", contains: deep, maxContains: 1 },
},
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const properties = fitted.properties as Record<string, any>;
expect(properties.exactly_one.oneOf).toBeUndefined();
expect(properties.exactly_one.anyOf).toHaveLength(2);
expect(properties.excluded).not.toHaveProperty("not");
expect(properties.conditional).not.toHaveProperty("if");
expect(properties.conditional).not.toHaveProperty("then");
expect(properties.bounded).not.toHaveProperty("maxContains");
const valid = instance(3, 7);
expectWidened(schema, fitted, [
{ exactly_one: { field_0: valid.field_0 } },
{ exactly_one: { field_1: valid.field_1 } },
{ excluded: { other: true } },
{ conditional: { field_1: valid.field_1 } },
{ conditional: valid },
{ bounded: [valid, { field_0: 1 }] },
]);
});
it("advertises items only beside the single array type", () => {
const rows = {
type: "array",
items: { type: "object", properties: { id: { type: "string" } } },
minItems: 1,
description: "Rows.",
};
const schema = {
type: "object",
properties: {
// Its large leaf one level deeper limits the retained depth to two.
large: {
type: "object",
properties: {
inner: {
type: "object",
properties: {
text: { type: "string", description: "x".repeat(70_000) },
},
},
},
},
arrays: {
type: "object",
properties: {
rows,
optional_rows: { anyOf: [rows, { type: "null" }] },
cube: { type: "array", items: { type: "array", items: rows } },
},
},
},
};
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect((fitted.properties as Record<string, any>).arrays).toEqual({
type: "object",
properties: {
rows: {
type: "array",
items: { type: "object" },
minItems: 1,
description: "Rows.",
},
optional_rows: {
anyOf: [{ type: "array", items: { type: "object" } }, { type: "null" }],
},
cube: { type: "array", items: {} },
},
});
expectWidened(schema, fitted, [
{
arrays: {
rows: [{ id: "a" }],
optional_rows: null,
cube: [[[{ id: "c" }]]],
},
},
{ arrays: { optional_rows: [{ id: "b" }] } },
]);
});
it("falls back to the compact projection for schemas with local references", () => {
const schema = {
type: "object",
$defs: { capture: record(3, 12) },
properties: {
before: { $ref: "#/$defs/capture" },
after: { $ref: "#/$defs/capture" },
},
};
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(JSON.stringify(fitted)).not.toContain("$ref");
expect(ajv().validateSchema(fitted)).toBe(true);
});
it("presents the compact profile from the complete projection", () => {
const capture = record(3, 7);
const complete = {
type: "object",
properties: {
before: { ...capture, description: "Earlier capture." },
after: { ...capture, description: "Later capture." },
},
};
const fitted = fitAdvertisedInputSchema(complete);
expect(fitted).not.toEqual(complete);
const budget = { budgetBytes: 13 * 1024 };
expect(presentAdvertisedInputProfile(fitted, budget)).toEqual(
presentAdvertisedInputSchema(complete, budget),
);
expect(presentAdvertisedInputProfile(fitted, undefined)).toBe(fitted);
});
describe("advertised numeric literals", () => {
it("relaxes numeric const and enum members without narrowing", () => {
const schema = {
type: "object",
properties: {
version: { type: "integer", const: 2 },
ratio: { const: 0.5 },
level: { enum: [1, 2, 3] },
mixed: { enum: ["auto", 7] },
not_two: { type: "integer", not: { const: 2 } },
},
};
const presented = presentInputJsonSchema(schema, () => undefined);
expect(presented.properties).toEqual({
version: { type: "integer" },
ratio: { type: "number" },
level: { type: "integer" },
mixed: { type: ["string", "integer"] },
not_two: { type: "integer" },
});
expectWidened(schema, presented, [
{ version: 2, ratio: 0.5, level: 3, mixed: "auto", not_two: 3 },
{ mixed: 7 },
]);
});
it("drops unevaluated keywords when relaxing an applicator they depend on", () => {
const schema = {
type: "object",
properties: {
options: {
type: "object",
properties: { mode: { type: "integer" } },
if: { properties: { mode: { const: 2 } }, required: ["mode"] },
then: { properties: { extra: { type: "string" } } },
unevaluatedProperties: false,
},
},
};
const presented = presentInputJsonSchema(schema, () => undefined);
expect(presented.properties).toEqual({
options: { type: "object", properties: { mode: { type: "integer" } } },
});
expectWidened(schema, presented, [
{ options: { mode: 2, extra: "kept" } },
{ options: { mode: 1 } },
]);
});
});
+423 -46
View File
@@ -9,6 +9,11 @@ import {
schemaChildren,
schemaMaps,
} from "./jsonSchemaKeywords.js";
import {
compactAdvertisedInputSchema,
presentAdvertisedInputSchema,
type CompactInputSchemaPresentation,
} from "./compactInputSchema.js";
const isObject = (value: unknown): value is Readonly<Record<string, unknown>> =>
typeof value === "object" && value !== null && !Array.isArray(value);
@@ -155,6 +160,223 @@ type SchemaNormalization = {
readonly changed: boolean;
};
const serializedBytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
/** Instance nesting levels below the current node that keep their structure. */
type SchemaSummary = { readonly levels: number };
// Keywords whose subschemas describe a nested instance. Combinators,
// conditionals and dependent schemas describe the same instance as their parent.
const instanceChildKeys = new Set([
"properties",
"patternProperties",
"additionalProperties",
"unevaluatedProperties",
"propertyNames",
"prefixItems",
"items",
"additionalItems",
"contains",
"unevaluatedItems",
]);
const hasNestedSchema = (schema: Readonly<Record<string, unknown>>): boolean =>
Object.entries(schema).some(([key, child]) =>
schemaMaps.has(key)
? isObject(child) && Object.values(child).some(isObject)
: schemaArrays.has(key)
? Array.isArray(child) && child.length > 0
: schemaChildren.has(key) && isObject(child),
);
/** Count nested instance levels a schema describes, ignoring literal data. */
const schemaLevels = (schema: unknown): number => {
if (!isObject(schema)) return 0;
let deepest = 0;
for (const [key, child] of Object.entries(schema)) {
const children = schemaMaps.has(key)
? isObject(child)
? Object.values(child)
: []
: schemaArrays.has(key)
? Array.isArray(child)
? child
: []
: schemaChildren.has(key)
? [child]
: [];
const step = instanceChildKeys.has(key) ? 1 : 0;
for (const nested of children)
if (isObject(nested))
deepest = Math.max(deepest, step + schemaLevels(nested));
}
return deepest;
};
const literalType = (value: unknown): string => {
if (value === null) return "null";
if (Array.isArray(value)) return "array";
if (typeof value === "number")
return Number.isInteger(value) ? "integer" : "number";
return typeof value;
};
const typeKeyword = (types: readonly string[]): string | string[] =>
types.length === 1 ? types[0]! : [...types];
/** Every JSON type a schema can accept, or undefined when that is not explicit. */
const schemaTypes = (schema: unknown): readonly string[] | undefined => {
if (!isObject(schema)) return undefined;
if (typeof schema.type === "string") return [schema.type];
if (
Array.isArray(schema.type) &&
schema.type.length > 0 &&
schema.type.every((type) => typeof type === "string")
)
return schema.type;
const literals = Object.hasOwn(schema, "const")
? [schema.const]
: Array.isArray(schema.enum) && schema.enum.length > 0
? schema.enum
: undefined;
if (literals !== undefined) return [...new Set(literals.map(literalType))];
const branches = Array.isArray(schema.anyOf)
? schema.anyOf
: Array.isArray(schema.oneOf)
? schema.oneOf
: [];
const types = branches.map(schemaTypes);
return types.length > 0 && types.every((type) => type !== undefined)
? [...new Set(types.flat())]
: undefined;
};
/** Item types of an explicitly typed array, including tuple positions. */
const typedArrayItemTypes = (
schema: Readonly<Record<string, unknown>>,
): readonly string[] | undefined => {
// Tuple positions come from `prefixItems` (2020-12) or an `items` array
// (draft-07); later elements from `items` or `additionalItems`.
const positions = [
...(Array.isArray(schema.prefixItems) ? schema.prefixItems : []),
...(Array.isArray(schema.items) ? schema.items : []),
];
const rest = Array.isArray(schema.items)
? schema.additionalItems
: schema.items;
// An absent or `true` rest schema admits any later element.
if (rest !== false && !isObject(rest)) return undefined;
const types = [...positions, ...(rest === false ? [] : [rest])].map(
schemaTypes,
);
return types.every((type) => type !== undefined)
? [...new Set(types.flat())]
: undefined;
};
/** Every type an array accepted by the schema can contain, when explicit. */
const arrayItemTypes = (schema: unknown): readonly string[] | undefined => {
if (!isObject(schema)) return undefined;
const types = schemaTypes(schema);
if (types !== undefined && !types.includes("array")) return [];
if (schema.type !== undefined) return typedArrayItemTypes(schema);
const branches = Array.isArray(schema.anyOf)
? schema.anyOf
: Array.isArray(schema.oneOf)
? schema.oneOf
: [];
const itemTypes = branches.map(arrayItemTypes);
return itemTypes.length > 0 && itemTypes.every((type) => type !== undefined)
? [...new Set(itemTypes.flat())]
: undefined;
};
// Size constraints on the summarized instance itself. Required names stay
// with the properties they name: Gemini rejects required undeclared properties.
const summarizedConstraintKeys = [
"minProperties",
"maxProperties",
"minItems",
"maxItems",
] as const;
/**
* Replace a nested schema with its accepted types, own size constraints and
* guidance. The summary accepts every value the complete
* schema accepts, so it only widens the advertisement; `items` appears only
* beside the single `array` type.
*/
export const summarizedSchema = (
schema: Readonly<Record<string, unknown>>,
): Record<string, unknown> => {
const retained = {
...Object.fromEntries(
summarizedConstraintKeys.flatMap((key) =>
Object.hasOwn(schema, key) ? [[key, schema[key]]] : [],
),
),
...(typeof schema.description === "string"
? { description: schema.description }
: {}),
};
const types = schemaTypes(schema);
if (types === undefined) return retained;
const others = types.filter((type) => type !== "array");
if (others.length === types.length)
return { type: typeKeyword(types), ...retained };
const itemTypes = arrayItemTypes(schema);
const array = {
type: "array",
items:
itemTypes === undefined ||
itemTypes.length === 0 ||
itemTypes.includes("array")
? {}
: { type: typeKeyword(itemTypes) },
};
return others.length === 0
? { ...array, ...retained }
: { anyOf: [array, { type: typeKeyword(others) }], ...retained };
};
const withoutDescription = ({
description: _description,
...schema
}: Readonly<Record<string, unknown>>): Readonly<Record<string, unknown>> =>
schema;
/**
* Summarize a property whose nested schema repeats an earlier sibling's, such
* as `after` beside `before`, as a pointer to that sibling. Records the first
* occurrence of each nested schema in `complete`.
*/
const sharedSiblingSchema = (
name: string,
child: unknown,
complete: [string, Readonly<Record<string, unknown>>][],
): Record<string, unknown> | undefined => {
if (!isObject(child) || !hasNestedSchema(child)) return undefined;
const shape = withoutDescription(child);
const sibling = complete.find(([, known]) =>
isDeepStrictEqual(withoutDescription(known), shape),
)?.[0];
if (sibling === undefined) {
complete.push([name, child]);
return undefined;
}
const shared = {
...summarizedSchema(child),
description: [
typeof child.description === "string" ? child.description : undefined,
`Same schema as \`${sibling}\`.`,
]
.filter((text) => text !== undefined)
.join(" "),
};
return serializedBytes(shared) < serializedBytes(child) ? shared : undefined;
};
const isBooleanLiteralSchema = (schema: unknown): boolean =>
isObject(schema) &&
(typeof schema.const === "boolean" ||
@@ -182,36 +404,122 @@ const preserveSharedDefault = (
schema.default = defaults[0];
};
const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
/**
* Relax literal constraints that Gemini declarations reject, in place, without
* rejecting a value the literal accepted. Returns whether anything changed.
*/
const relaxUnsupportedLiterals = (node: Record<string, unknown>): boolean => {
let relaxed = false;
if (typeof node.const === "boolean") {
delete node.const;
if (node.type === undefined) node.type = "boolean";
relaxed = true;
}
if (node.const === "") {
delete node.const;
if (node.type === undefined) node.type = "string";
relaxed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.length > 0 &&
node.enum.every((item) => typeof item === "boolean")
) {
delete node.enum;
if (node.type === undefined) node.type = "boolean";
relaxed = true;
}
if (Array.isArray(node.enum) && node.enum.includes("")) {
// Removing only the empty member would reject a canonical-valid input.
// Gemini transports turn const into enum and refuse empty enum members.
const values = node.enum;
delete node.enum;
if (
node.type === undefined &&
values.every((item) => typeof item === "string")
)
node.type = "string";
relaxed = true;
}
if (typeof node.const === "number") {
// Gemini accepts only string enum members, and transports turn const into
// enum. The declared or literal numeric type still guides callers.
const literal = node.const;
delete node.const;
if (node.type === undefined) node.type = literalType(literal);
relaxed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.some((item) => typeof item === "number")
) {
const values = node.enum;
delete node.enum;
if (node.type === undefined)
node.type = typeKeyword([...new Set(values.map(literalType))]);
relaxed = true;
}
return relaxed;
};
const normalizeAdvertisedSchema = (
value: unknown,
summary?: SchemaSummary,
): SchemaNormalization => {
if (!isObject(value)) return { schema: value, changed: false };
if (summary !== undefined && summary.levels <= 0 && hasNestedSchema(value)) {
// A summary can be longer than the schema it replaces, such as a nullable
// array presented as anyOf. Keep the complete schema then.
const summarized = summarizedSchema(value);
return serializedBytes(summarized) < serializedBytes(value)
? { schema: summarized, changed: true }
: normalizeAdvertisedSchema(value);
}
const nested = (key: string): SchemaSummary | undefined =>
summary !== undefined && instanceChildKeys.has(key)
? { levels: summary.levels - 1 }
: summary;
const node: Record<string, unknown> = { ...value };
let changed = false;
const normalizeMap = (map: Readonly<Record<string, unknown>>) =>
Object.fromEntries(
const normalizeMap = (
key: string,
map: Readonly<Record<string, unknown>>,
) => {
const complete: [string, Readonly<Record<string, unknown>>][] = [];
return Object.fromEntries(
Object.entries(map).map(([name, child]) => {
const result = normalizeAdvertisedSchema(child);
const shared =
summary !== undefined && key === "properties"
? sharedSiblingSchema(name, child, complete)
: undefined;
if (shared !== undefined) {
changed = true;
return [name, shared];
}
const result = normalizeAdvertisedSchema(child, nested(key));
changed ||= result.changed;
return [name, result.schema];
}),
);
const normalizeList = (list: readonly unknown[]) =>
};
const normalizeList = (key: string, list: readonly unknown[]) =>
list.map((child) => {
const result = normalizeAdvertisedSchema(child);
const result = normalizeAdvertisedSchema(child, nested(key));
changed ||= result.changed;
return result;
});
for (const [key, child] of Object.entries(value)) {
if (schemaMaps.has(key) && isObject(child)) {
node[key] = normalizeMap(child);
node[key] = normalizeMap(key, child);
} else if (
schemaArrays.has(key) &&
key !== "anyOf" &&
key !== "oneOf" &&
Array.isArray(child)
) {
node[key] = normalizeList(child).map((result) => result.schema);
node[key] = normalizeList(key, child).map((result) => result.schema);
} else if (
schemaChildren.has(key) &&
key !== "not" &&
@@ -220,8 +528,8 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
key !== "else"
) {
const results = Array.isArray(child)
? normalizeList(child)
: [normalizeAdvertisedSchema(child)];
? normalizeList(key, child)
: [normalizeAdvertisedSchema(child, nested(key))];
const nestedChanged = results.some((result) => result.changed);
if (
key === "contains" &&
@@ -241,40 +549,10 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
}
}
if (typeof node.const === "boolean") {
delete node.const;
if (node.type === undefined) node.type = "boolean";
changed = true;
}
if (node.const === "") {
delete node.const;
if (node.type === undefined) node.type = "string";
changed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.length > 0 &&
node.enum.every((item) => typeof item === "boolean")
) {
delete node.enum;
if (node.type === undefined) node.type = "boolean";
changed = true;
}
if (Array.isArray(node.enum) && node.enum.includes("")) {
// Removing only the empty member would reject a canonical-valid input.
// Gemini transports turn const into enum and refuse empty enum members.
const values = node.enum;
delete node.enum;
if (
node.type === undefined &&
values.every((item) => typeof item === "string")
)
node.type = "string";
changed = true;
}
if (relaxUnsupportedLiterals(node)) changed = true;
const negated = Object.hasOwn(value, "not")
? normalizeAdvertisedSchema(value.not)
? normalizeAdvertisedSchema(value.not, summary)
: undefined;
if (negated?.changed) {
// Widening a predicate under `not` narrows the containing schema.
@@ -285,13 +563,13 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
}
const conditional = Object.hasOwn(value, "if")
? normalizeAdvertisedSchema(value.if)
? normalizeAdvertisedSchema(value.if, summary)
: undefined;
const thenBranch = Object.hasOwn(value, "then")
? normalizeAdvertisedSchema(value.then)
? normalizeAdvertisedSchema(value.then, summary)
: undefined;
const elseBranch = Object.hasOwn(value, "else")
? normalizeAdvertisedSchema(value.else)
? normalizeAdvertisedSchema(value.else, summary)
: undefined;
if (conditional?.changed) {
// Changing the predicate can route canonical-valid inputs into a rejecting branch.
@@ -314,7 +592,7 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
for (const combinator of ["anyOf", "oneOf"] as const) {
const branches = value[combinator];
if (!Array.isArray(branches)) continue;
const normalized = normalizeList(branches);
const normalized = normalizeList(combinator, branches);
const schemas = normalized.map((result) => result.schema);
const branchesChanged = normalized.some((result) => result.changed);
if (combinator === "oneOf" && branchesChanged) {
@@ -353,6 +631,13 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
changed ||= branchesChanged;
}
if (changed) {
// A relaxed or summarized applicator can stop evaluating properties or
// items, which these keywords would then reject.
delete node.unevaluatedProperties;
delete node.unevaluatedItems;
}
return { schema: node, changed };
};
@@ -623,3 +908,95 @@ export const presentInputJsonSchema = (
description: description.join(" "),
}).schema as Record<string, unknown>;
};
/**
* Claude Desktop drops a local connector tool whose input schema, serialized
* as compact JSON, exceeds 64 KiB (#4081). Other clients take the same
* catalog, so every advertised input schema fits this limit.
*/
export const ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES = 64 * 1024;
const SUMMARIZED_GUIDANCE =
"Deeply nested and repeated inputs are advertised in summarized form to fit client schema-size limits; the server still validates complete canonical inputs.";
// The compact profile starts from the complete self-contained projection,
// not from a fitted summary of it.
const completeProjections = new WeakMap<object, Record<string, unknown>>();
/**
* Return the input schema clients receive for the selected profile: the
* fitted full advertisement, or the compact presentation of its complete
* projection.
*/
export const presentAdvertisedInputProfile = (
advertised: Record<string, unknown>,
compactPresentation: CompactInputSchemaPresentation | undefined,
): Record<string, unknown> =>
compactPresentation === undefined
? advertised
: presentAdvertisedInputSchema(
completeProjections.get(advertised) ?? advertised,
compactPresentation,
);
/**
* Fit an advertised input schema within the client size limit. A schema that
* already fits is returned unchanged. Otherwise identical sibling properties
* point at the first one, and nested schemas below the deepest level that
* still fits are summarized as their accepted types and guidance. Every
* rewrite widens the advertisement, so canonical inputs and examples remain
* accepted; the registered canonical schema still validates every call.
*/
export const fitAdvertisedInputSchema = (
schema: Record<string, unknown>,
): Record<string, unknown> => {
if (serializedBytes(schema) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES)
return schema;
// Summaries remove subschemas that a local reference could target.
const references = new JsonSchemaReferences(schema).references();
if (references.next().done !== true) return fitByCompaction(schema);
const described = {
...schema,
description: [
typeof schema.description === "string" ? schema.description : undefined,
SUMMARIZED_GUIDANCE,
]
.filter((text) => text !== undefined)
.join(" "),
};
// Size need not grow with the retained depth: a deeper candidate can be
// smaller than a summary. Keep the deepest candidate that fits.
for (let levels = schemaLevels(schema); levels >= 1; levels -= 1) {
const candidate = summarizedInputSchemaCandidate(described, levels);
if (serializedBytes(candidate) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES) {
completeProjections.set(candidate, schema);
return candidate;
}
}
return fitByCompaction(schema);
};
/**
* Advertise identical siblings as pointers and summarize schemas below the
* given instance level. Exposed so tests can compare fitting candidates.
*/
export const summarizedInputSchemaCandidate = (
schema: Record<string, unknown>,
levels: number,
): Record<string, unknown> =>
normalizeAdvertisedSchema(schema, { levels }).schema as Record<
string,
unknown
>;
/** Fall back to the compact projection, which fits any budget. */
const fitByCompaction = (
schema: Record<string, unknown>,
): Record<string, unknown> => {
const { schema: fitted } = compactAdvertisedInputSchema(
schema,
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
completeProjections.set(fitted, schema);
return fitted;
};
@@ -246,6 +246,7 @@ const nativeExportEvidence = () => {
exports: coverage([
{
name: "open_native",
runtime_name: "open_native",
address: "0x1000",
weak: false,
reexport: false,
+2 -2
View File
@@ -49,8 +49,8 @@ export const OFFICIAL_TOOL_CONTRACTS = [
),
official(
"inspect_native_load_image",
"Verify the provider's loaded DOS MZ or explicitly selected COM image against its immutable target snapshot: complete original/modified source-byte digests, header/module file mappings and memory digests, relocations and external entry. Returns measured observations and independent checks with mismatch coordinates. COM verification includes its imposed entry register context and does not claim PSP/runtime coverage. Overlay and uninitialized coverage remain explicit; other formats return unsupported. Does not execute or modify the target.",
z.strictObject({}),
"Inspect the provider's measured source digests, memory mappings, relocations, entry points and entry register context. Independent snapshot verification supports DOS MZ, explicitly selected COM, raw binaries and GameCube DOL; other formats retain their observations with unsupported verification. Overlay and uninitialized coverage remain explicit. Select detail summary for verification state and counts, retaining complete Evidence for inspect_analysis_view kind native-load-image: collection source_files, mappings, relocations, entry_points, entry_context, checks or sections, with indices, fields and offset/limit paging. Relocation bytes and check expected/observed payloads require explicit fields; defaults keep metadata. Null collections are unrecorded or inapplicable. Does not execute or modify the target.",
z.strictObject({ detail: analysisDetailSchema }),
),
official(
"inspect_native_data_type",
+23
View File
@@ -1,8 +1,31 @@
import { z } from "zod";
import { absoluteLocalPathStringSchema } from "../domain/localPath.js";
/** Caller-selected binary_session projections. */
export const BINARY_SESSION_DETAILS = ["summary", "complete"] as const;
export type BinarySessionDetail = (typeof BINARY_SESSION_DETAILS)[number];
/**
* Fields a summary leaves out, as dotted paths through arrays. Each stays
* available from the same session with detail: "complete".
*/
export const BINARY_SESSION_SUMMARY_OMITTED_FIELDS = [
"analysis_provider_candidates.availability.diagnostics",
"analysis_provider_candidates.target_support.diagnostics",
"analysis_provider_candidates.capabilities",
"tool_availability.groups.surface",
"tool_availability.groups.effects",
"tool_availability.groups.annotations",
] as const;
/** Optional expectations used to compare the live session with its caller. */
export const binarySessionInputSchema = z.strictObject({
detail: z
.enum(BINARY_SESSION_DETAILS)
.default("summary")
.describe(
"summary (default) keeps open state, the binding, current-route capabilities, each provider candidate's availability and target-support status, code and reason, and every tool's availability, reason and remediation; omitted_fields lists what it leaves out. complete adds provider diagnostics, candidate capabilities and static tool surface, effect and annotation flags.",
),
expected_package_version: z.string().min(1).optional(),
expected_catalog_digest: z
.string()
+1 -1
View File
@@ -80,7 +80,7 @@ export const SESSION_TOOL_CONTRACTS = [
),
session(
"binary_session",
"Report the active binary target or resource-only app root, provider and capability availability when applicable, client-feature, analysis, and server-identity status without starting analysis.",
'Report the active binary target or resource-only app root, provider and capability availability when applicable, client-feature, analysis, and server-identity status without starting analysis. The default summary lists omitted_fields; detail: "complete" adds provider diagnostics, candidate capabilities and static tool flags.',
binarySessionInputSchema,
),
session(
+36 -22
View File
@@ -5,6 +5,7 @@ import {
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
artifactInventorySummaryViewResultSchema,
interfaceBuilderSummaryViewResultSchema,
machoSummaryViewResultSchema,
@@ -14,7 +15,15 @@ import {
import { nativeFunctionMatchingSchema } from "../domain/native/functionMatching.js";
import { nativeByteReadSchema } from "../domain/native/nativeByteRead.js";
import { analysisDetailOutputSchemaOf } from "./analysisDetail.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import {
functionDossierSchema,
analysisBookmarkSchema,
functionInstructionWindowSchema,
referenceEdgeSchema,
unresolvedCallSchema,
procedureIdentitySchema,
containingProcedureResolutionSchema,
} from "../domain/hopperValues.js";
import { nativeInventoryRowsSchemas } from "../domain/native/nativeInventory.js";
import { z } from "zod";
import { byteSearchResultSchema } from "../domain/native/bytePattern.js";
@@ -43,14 +52,6 @@ import { residualUnknownSchema } from "../domain/residualUnknown.js";
import { evidenceBundleSchema } from "../domain/evidenceBundle.js";
import { processCaptureComparisonSchema } from "../domain/process/processComparison.js";
import {
analysisBookmarkSchema,
functionInstructionWindowSchema,
referenceEdgeSchema,
unresolvedCallSchema,
procedureIdentitySchema,
containingProcedureResolutionSchema,
} from "../domain/hopperValues.js";
import { nativeApiInspectionResultSchema } from "../domain/native/nativeApiBoundary.js";
import {
demangleSwiftSchema,
@@ -97,7 +98,8 @@ import {
objcClassDiscoveryOutput,
evidenceResultOf,
segmentOutput,
sessionProvider,
sessionProviderFacts,
sessionStatusDetailShapes,
symbolDiscoveryOutput,
targetFormatSchema,
targetKindSchema,
@@ -129,7 +131,10 @@ const bookmarkFacetSchema = z.discriminatedUnion("state", [
export const officialOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
search_bytes: evidenceResultOf(byteSearchResultSchema),
annotate_native_function: evidenceResultOf(nativeFunctionAnnotationsSchema),
inspect_native_load_image: evidenceResultOf(nativeLoadImageSchema),
inspect_native_load_image: analysisDetailOutputSchemaOf(
nativeLoadImageSchema,
nativeLoadImageSummaryViewResultSchema,
),
inspect_native_data_type: evidenceResultOf(nativeDataTypeSchema),
inspect_native_instruction: evidenceResultOf(nativeInstructionSchema),
resolve_native_call_targets: evidenceResultOf(nativeCallTargetsSchema),
@@ -564,18 +569,27 @@ export const sessionOutputSchemas = {
}),
]),
),
// Every session kind crossed with every detail level, so `detail` fixes
// which candidate, tool-group and omission fields the result carries.
binary_session: lifecycleResultOf(
z.union([
sessionProvider.extend({ open: z.literal(false) }),
sessionProvider.extend({
open: z.literal(true),
session_kind: z.literal("bundle_resources"),
resource_root: bundleResourceRootSchema,
}),
...openLifecycleTargetVariants.map((variant) =>
sessionProvider.extend({ open: z.literal(true), ...variant.shape }),
),
]),
z.union(
sessionStatusDetailShapes.flatMap((detail) => [
sessionProviderFacts.extend({ open: z.literal(false), ...detail }),
sessionProviderFacts.extend({
open: z.literal(true),
session_kind: z.literal("bundle_resources"),
resource_root: bundleResourceRootSchema,
...detail,
}),
...openLifecycleTargetVariants.map((variant) =>
sessionProviderFacts.extend({
open: z.literal(true),
...variant.shape,
...detail,
}),
),
]),
),
),
export_evidence_bundle: lifecycleResultOf(
z.object({
+155 -43
View File
@@ -21,6 +21,7 @@ import { analysisErrorProjectionSchema } from "./errorSchemas.js";
import { relationshipCoverageSchema } from "../domain/relationshipCoverage.js";
import { ZIP_PACKAGE_FORMATS } from "../domain/zipPackageFormat.js";
import { objcClassImportIdentitySchema } from "../domain/native/objcSwiftMetadata.js";
import { BINARY_SESSION_SUMMARY_OMITTED_FIELDS } from "./sessionStatusContract.js";
/** Specialize the single normalized result inside a complete Evidence record. */
export const evidenceResultOf = <Schema extends z.ZodType>(schema: Schema) =>
@@ -256,41 +257,48 @@ const providerRejectionCode: z.ZodType<ProviderRejectionCode> = z.enum(
const providerDiagnostics = z.record(z.string(), jsonValueSchema);
const providerAvailability = z.discriminatedUnion("status", [
z.object({
const providerAvailabilityStates = {
unknown: z.object({
status: z.literal("unknown"),
code: z.null(),
reason: z.string().min(1),
diagnostics: providerDiagnostics,
}),
z.object({
available: z.object({
status: z.literal("available"),
code: z.null(),
reason: z.null(),
diagnostics: providerDiagnostics,
}),
z.object({
unavailable: z.object({
status: z.literal("unavailable"),
code: providerRejectionCode,
reason: z.string().min(1),
}),
};
const providerAvailability = z.discriminatedUnion("status", [
providerAvailabilityStates.unknown.extend({
diagnostics: providerDiagnostics,
}),
providerAvailabilityStates.available.extend({
diagnostics: providerDiagnostics,
}),
providerAvailabilityStates.unavailable.extend({
diagnostics: providerDiagnostics,
}),
]);
const providerTargetSupport = z.discriminatedUnion("status", [
z.object({
const providerTargetSupportStates = {
unknown: z.object({
status: z.literal("unknown"),
code: z.null(),
reason: z.string().min(1),
diagnostics: providerDiagnostics,
}),
z.object({
supported: z.object({
status: z.literal("supported"),
code: z.null(),
reason: z.null(),
diagnostics: providerDiagnostics,
}),
z.object({
unsupported: z.object({
status: z.literal("unsupported"),
code: z.enum([
"target_kind_unsupported",
@@ -300,10 +308,46 @@ const providerTargetSupport = z.discriminatedUnion("status", [
"managed_target_unsupported",
]),
reason: z.string().min(1),
}),
};
const providerTargetSupport = z.discriminatedUnion("status", [
providerTargetSupportStates.unknown.extend({
diagnostics: providerDiagnostics,
}),
providerTargetSupportStates.supported.extend({
diagnostics: providerDiagnostics,
}),
providerTargetSupportStates.unsupported.extend({
diagnostics: providerDiagnostics,
}),
]);
/** Complete candidate facts, including provider-owned diagnostics and capability sets. */
export const analysisProviderCandidate = z.object({
provider: providerIdentity,
availability: providerAvailability,
target_support: providerTargetSupport,
selected: z.boolean(),
capabilities: z.array(providerCapabilitySet),
});
/** Summary candidate: the same status, code, and reason without diagnostics or capabilities. */
export const analysisProviderCandidateSummary = z.strictObject({
provider: providerIdentity,
availability: z.discriminatedUnion("status", [
providerAvailabilityStates.unknown.strict(),
providerAvailabilityStates.available.strict(),
providerAvailabilityStates.unavailable.strict(),
]),
target_support: z.discriminatedUnion("status", [
providerTargetSupportStates.unknown.strict(),
providerTargetSupportStates.supported.strict(),
providerTargetSupportStates.unsupported.strict(),
]),
selected: z.boolean(),
});
const clientFeatureNameSchema = z.enum([
"elicitation_form",
"elicitation_url",
@@ -404,45 +448,93 @@ export const TOOL_ANNOTATION_NAMES = Object.keys(
toolAvailabilityFacts.annotations.shape,
) as (keyof z.output<typeof toolAvailabilityFacts.annotations>)[];
const toolAvailabilityGroupFacts = {
const toolAvailabilityStatusGroupFacts = {
names: z.array(z.string().min(1)).min(1),
surface: toolAvailabilityFacts.surface,
default_mode_available: toolAvailabilityFacts.default_mode_available,
modes: toolAvailabilityFacts.modes,
client_requirements: toolAvailabilityFacts.client_requirements.optional(),
};
const toolAvailabilityGroupFacts = {
names: toolAvailabilityStatusGroupFacts.names,
surface: toolAvailabilityFacts.surface,
default_mode_available:
toolAvailabilityStatusGroupFacts.default_mode_available,
modes: toolAvailabilityStatusGroupFacts.modes,
client_requirements: toolAvailabilityStatusGroupFacts.client_requirements,
effects: z.array(toolAvailabilityFacts.effects.keyof()),
annotations: z.array(toolAvailabilityFacts.annotations.keyof()),
};
const toolAvailabilityGroup = z.discriminatedUnion("available", [
z.strictObject({
...toolAvailabilityGroupFacts,
available: z.literal(true),
reason: z.literal("available"),
remediation: z.null(),
}),
z.strictObject({
...toolAvailabilityGroupFacts,
available: z.literal(false),
reason: toolUnavailabilityReason,
remediation: z.string().nullable(),
}),
]);
const toolAvailabilityGroupOf = <Facts extends z.ZodRawShape>(facts: Facts) =>
z.discriminatedUnion("available", [
z.strictObject({
...facts,
available: z.literal(true),
reason: z.literal("available"),
remediation: z.null(),
}),
z.strictObject({
...facts,
available: z.literal(false),
reason: toolUnavailabilityReason,
remediation: z.string().nullable(),
}),
]);
/** Complete inventory with explicit inherited client requirements and enabled flags. */
export const toolAvailabilityInventory = z.strictObject({
client_requirements: toolAvailabilityFacts.client_requirements,
groups: z.array(toolAvailabilityGroup),
groups: z.array(toolAvailabilityGroupOf(toolAvailabilityGroupFacts)),
});
export type ToolAvailabilityInventory = z.output<
typeof toolAvailabilityInventory
>;
/**
* Summary inventory: every tool name with its current availability, reason,
* remediation, modes, and client requirements, grouped by those facts alone.
* Static surface, effect, and annotation flags are omitted, not false.
*/
export const toolAvailabilitySummaryInventory = z.strictObject({
client_requirements: toolAvailabilityFacts.client_requirements,
groups: z.array(toolAvailabilityGroupOf(toolAvailabilityStatusGroupFacts)),
});
export type ToolAvailabilitySummaryInventory = z.output<
typeof toolAvailabilitySummaryInventory
>;
/** One tool's current availability without its static catalog flags. */
export type ToolAvailabilitySummary = Omit<
ToolAvailability,
"surface" | "effects" | "annotations"
>;
const isCompleteToolAvailabilityInventory = (
inventory: ToolAvailabilityInventory | ToolAvailabilitySummaryInventory,
): inventory is ToolAvailabilityInventory =>
inventory.groups.every((group) => "surface" in group);
/** Expand the wire representation once at the consuming boundary. */
export const expandToolAvailability = (
export function expandToolAvailability(
inventory: ToolAvailabilityInventory,
): readonly ToolAvailability[] =>
inventory.groups.flatMap((group) =>
): readonly ToolAvailability[];
export function expandToolAvailability(
inventory: ToolAvailabilityInventory | ToolAvailabilitySummaryInventory,
): readonly (ToolAvailability | ToolAvailabilitySummary)[];
export function expandToolAvailability(
inventory: ToolAvailabilityInventory | ToolAvailabilitySummaryInventory,
): readonly (ToolAvailability | ToolAvailabilitySummary)[] {
if (!isCompleteToolAvailabilityInventory(inventory))
return inventory.groups.flatMap(({ names, ...group }) =>
names.map((name) => ({
...group,
name,
client_requirements:
group.client_requirements ?? inventory.client_requirements,
})),
);
return inventory.groups.flatMap((group) =>
group.names.map((name) =>
toolAvailability.parse({
...group,
@@ -464,8 +556,10 @@ export const expandToolAvailability = (
}),
),
);
}
export const sessionProvider = z
/** Session status facts shared by every detail level and session kind. */
export const sessionProviderFacts = z
.object({
cleanup_pending: z.boolean(),
capabilities: z.array(providerCapabilitySet),
@@ -483,16 +577,6 @@ export const sessionProvider = z
analysis_profile: analysisProfileSchema,
})
.nullable(),
analysis_provider_candidates: z.array(
z.object({
provider: providerIdentity,
availability: providerAvailability,
target_support: providerTargetSupport,
selected: z.boolean(),
capabilities: z.array(providerCapabilitySet),
}),
),
tool_availability: toolAvailabilityInventory,
client_features: z.object({
elicitation_form: z.boolean(),
elicitation_url: z.boolean(),
@@ -546,6 +630,34 @@ export const sessionProvider = z
})
.passthrough();
/**
* Fields whose shape `detail` selects. A summary must list what it omits and
* carry only summary candidates and tool groups; complete detail must carry
* diagnostics, candidate capabilities and static tool flags, and no omission list.
*/
export const sessionStatusDetailShapes = [
{
detail: z.literal("summary"),
omitted_fields: z
.array(z.enum(BINARY_SESSION_SUMMARY_OMITTED_FIELDS))
.min(1),
analysis_provider_candidates: z.array(analysisProviderCandidateSummary),
tool_availability: toolAvailabilitySummaryInventory,
},
{
detail: z.literal("complete"),
omitted_fields: z.never().optional(),
analysis_provider_candidates: z.array(analysisProviderCandidate),
tool_availability: toolAvailabilityInventory,
},
] as const;
/** Session status for either detail level, keyed by `detail`. */
export const sessionProvider = z.discriminatedUnion("detail", [
sessionProviderFacts.extend(sessionStatusDetailShapes[0]),
sessionProviderFacts.extend(sessionStatusDetailShapes[1]),
]);
export const nullableText = z.string().nullable();
export const addressList = z.array(z.string());
export const addressedEntry = z.object({
+9 -6
View File
@@ -6,7 +6,10 @@ import { z } from "zod";
import type { ToolContract } from "./toolContractTypes.js";
import { advertisedInputExamples } from "./advertisedInputExamples.js";
import { presentInputJsonSchema } from "./inputSchemaPresentation.js";
import {
fitAdvertisedInputSchema,
presentInputJsonSchema,
} from "./inputSchemaPresentation.js";
import {
schemaArrays,
schemaChildren,
@@ -75,13 +78,13 @@ export const toolInputSchemaWithMetadata = <Contract extends ToolContract>(
libraryOptions: { ...options.libraryOptions, reused: "inline" },
});
const examples = advertisedInputExamples(contract);
return {
// Keep advertised parameters self-contained for clients that do not
// resolve local references. The registered canonical schema remains
// responsible for runtime validation.
// Keep advertised parameters self-contained for clients that do not
// resolve local references, within the per-tool client size limit. The
// registered canonical schema remains responsible for runtime validation.
return fitAdvertisedInputSchema({
...presentInputJsonSchema(inline, fallbackPropertyDescription),
...(examples.length === 0 ? {} : { examples }),
};
});
},
);
+9 -1
View File
@@ -108,14 +108,20 @@ export class AnalysisProtocolError extends AnalysisError {
export class AnalysisInputError extends AnalysisError {
readonly _tag = "AnalysisInputError";
override readonly userMessage: string | undefined;
/** Refusal-specific correction, when the generic input-contract advice cannot fix it. */
readonly remediationAction: string | undefined;
constructor(
readonly operation: string,
options?: AnalysisErrorOptions & { readonly userMessage?: string },
options?: AnalysisErrorOptions & {
readonly userMessage?: string;
readonly remediationAction?: string;
},
readonly issues: readonly AnalysisInputIssue[] = [],
) {
super(`Invalid analysis input for ${operation}`, options);
this.userMessage = options?.userMessage;
this.remediationAction = options?.remediationAction;
}
}
@@ -132,6 +138,8 @@ export interface AnalysisInputIssue {
/** Schema-authored correction guidance for cross-field or custom checks. */
readonly message?: string;
readonly expected?: JsonValue;
/** Producer-examined facts about the refused value, such as its location and shape. */
readonly observed?: JsonValue;
readonly minimum?: number;
readonly maximum?: number;
}
+4 -1
View File
@@ -161,7 +161,10 @@ export const analysisErrorRemediationAction = (
if (error instanceof ConfigurationError && error.settings.length > 0)
return `Correct ${[...new Set(error.settings.map(({ setting }) => setting))].join(", ")} in the environment that starts REA (your shell or the MCP client's registration), then rerun.`;
if (error instanceof AnalysisInputError)
return `Read the declared input schema or CLI usage for ${error.operation}; if using MCP and the catalog may be stale, call tools/list. Correct the listed fields, then retry. Do not repeat the unchanged call.`;
return (
error.remediationAction ??
`Read the declared input schema or CLI usage for ${error.operation}; if using MCP and the catalog may be stale, call tools/list. Correct the listed fields, then retry. Do not repeat the unchanged call.`
);
if (
error instanceof AnalysisOutputError &&
error.remediationAction !== undefined
+1
View File
@@ -368,6 +368,7 @@ const requestErrorDetails = (
reason: issue.reason,
...(issue.message === undefined ? {} : { message: issue.message }),
...(issue.expected === undefined ? {} : { expected: issue.expected }),
...(issue.observed === undefined ? {} : { observed: issue.observed }),
...(issue.minimum === undefined ? {} : { minimum: issue.minimum }),
...(issue.maximum === undefined ? {} : { maximum: issue.maximum }),
})),
+59 -6
View File
@@ -111,10 +111,10 @@ import {
import {
nativeInventoryViewRequestSchema,
nativeInventoryViewRowSchema,
nativeInventoryViewContext,
nativeInventorySummarySchema,
projectNativeInventoryView,
} from "./nativeInventoryView.js";
import { nativeViewSourceContext } from "./nativeViewContext.js";
import {
NATIVE_INVENTORY_OPERATIONS,
nativeInventoryOperationSchema,
@@ -122,6 +122,19 @@ import {
import { absoluteLocalPathStringSchema } from "../localPath.js";
import { z } from "zod";
import {
browserScenarioSummarySchema,
browserScenarioViewContextSchema,
browserScenarioViewItemsSchema,
browserScenarioViewRequestSchema,
} from "./browserScenarioViewSchemas.js";
import { projectBrowserScenarioView } from "./browserScenarioView.js";
import {
nativeLoadImageViewRequestSchema,
nativeLoadImageViewItemsSchema,
nativeLoadImageSummarySchema,
projectNativeLoadImageView,
} from "./nativeLoadImageView.js";
import {
desktopViewRequestSchema,
desktopViewItemsSchema,
@@ -380,6 +393,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
javascriptFactsViewRequestSchema,
nativeViewSchema,
nativeInventoryViewRequestSchema,
nativeLoadImageViewRequestSchema,
artifactInventoryViewRequestSchema,
wasmViewRequestSchema,
machoViewRequestSchema,
@@ -393,6 +407,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
environmentStateViewRequestSchema,
environmentCoverageViewRequestSchema,
captureViewRequestSchema,
browserScenarioViewRequestSchema,
protobufViewRequestSchema,
qlogViewRequestSchema,
pcapViewRequestSchema,
@@ -729,6 +744,14 @@ export const webNetworkCaptureSummaryViewResultSchema = z.strictObject({
summary: captureSummarySchema,
});
/** Compact outcomes and the focus screen from a retained browser scenario. */
export const browserScenarioSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
artifact: z.null(),
parent_operation: z.literal("capture_browser_scenario"),
summary: browserScenarioSummarySchema,
});
/** Exact package declarations with retained decoded manifest text. */
export const androidPackageSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
@@ -823,12 +846,21 @@ export const il2cppMetadataSummaryViewResultSchema = z.strictObject({
/** Exact inventory summary with the original observation's provider and query. */
export const nativeInventorySummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
...nativeInventoryViewContext,
...nativeViewSourceContext,
parent_operation: nativeInventoryOperationSchema,
artifact: artifactSchema.nullable(),
summary: nativeInventorySummarySchema,
});
/** Compact provider load observations retain the independent verification state. */
export const nativeLoadImageSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
...nativeViewSourceContext,
artifact: artifactSchema.nullable(),
parent_operation: z.literal("inspect_native_load_image"),
summary: nativeLoadImageSummarySchema,
});
/** Compact native matching discovery retains the complete matching report. */
export const nativeFunctionMatchingSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
@@ -1022,6 +1054,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
managedArtifactSummaryViewResultSchema,
managedMemberSummaryViewResultSchema,
webNetworkCaptureSummaryViewResultSchema,
browserScenarioSummaryViewResultSchema,
androidPackageSummaryViewResultSchema,
pcapCaptureSummaryViewResultSchema,
dartAotSummaryViewResultSchema,
@@ -1030,6 +1063,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
modelArtifactSummaryViewResultSchema,
gpuCodeObjectsSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
sourceToBundleSummaryViewResultSchema,
@@ -1054,13 +1088,23 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
}),
z.strictObject({
...viewResultBase,
...nativeInventoryViewContext,
...nativeViewSourceContext,
kind: z.literal("native-inventory"),
parent_operation: nativeInventoryOperationSchema,
artifact: artifactSchema.nullable(),
view: nativeInventoryViewRequestSchema,
rows: z.array(nativeInventoryViewRowSchema),
}),
z.strictObject({
...viewResultBase,
kind: z.literal("native-load-image"),
...nativeViewSourceContext,
verification_status: z.enum(["unsupported", "verified", "mismatch"]),
parent_operation: z.literal("inspect_native_load_image"),
artifact: artifactSchema.nullable(),
view: nativeLoadImageViewRequestSchema,
items: nativeLoadImageViewItemsSchema,
}),
z.strictObject({
kind: z.literal("protobuf"),
view: protobufViewRequestSchema,
@@ -1076,6 +1120,15 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
...viewResultBase,
parent_operation: z.literal("inspect_web_network_capture"),
}),
z.strictObject({
kind: z.literal("browser-scenario"),
view: browserScenarioViewRequestSchema,
items: browserScenarioViewItemsSchema,
...viewResultBase,
...browserScenarioViewContextSchema.shape,
artifact: z.null(),
parent_operation: z.literal("capture_browser_scenario"),
}),
z.strictObject({
kind: z.literal("qlog"),
view: qlogViewRequestSchema,
@@ -1283,9 +1336,7 @@ const unsupportedParent = (
new AnalysisUnsupportedTargetError(
INSPECT_ANALYSIS_VIEW_OPERATION,
parentArtifactPath(parent),
parent.operation === "capture_browser_scenario"
? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest, environment-state and environment-coverage views.`
: `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed Evidence from a supported parent operation with a compatible view. For a summary receipt, use its normalized_result.parent_evidence_id rather than the receipt's own evidence_id.`,
`Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed Evidence from a supported parent operation with a compatible view. For a summary receipt, use its normalized_result.parent_evidence_id rather than the receipt's own evidence_id.`,
);
const parentArtifactPath = (parent: AnalysisViewParent): string => {
@@ -1314,6 +1365,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
["inspect_erlang_beam", projectErlangBeamView],
["inspect_devicetree", projectDevicetreeView],
["inspect_native_exports", projectNativeExportsView],
["inspect_native_load_image", projectNativeLoadImageView],
["inspect_python_pyc", projectPythonPycView],
["frida_session_status", projectFridaSessionStatusView],
["gdb_console", projectCommandObservationView],
@@ -1331,6 +1383,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
["inspect_managed_artifact", projectManagedArtifactView],
["inspect_managed_members", projectManagedMemberView],
["inspect_android_package", projectAndroidPackageView],
["capture_browser_scenario", projectBrowserScenarioView],
["inspect_dart_aot", projectDartAotView],
["inspect_il2cpp_metadata", projectIl2cppMetadataView],
["inspect_apk_signature", projectApkSignatureView],
@@ -0,0 +1,193 @@
import { z } from "zod";
import type { RetainedBrowserScenarioCapture } from "../browserScenarioCapture.js";
import { isImmutableJsonSnapshot } from "../immutableJson.js";
import { compareAsciiStrings } from "../canonicalOrdering.js";
const count = z.int().nonnegative();
const statusClass = z.enum([
"1xx",
"2xx",
"3xx",
"4xx",
"5xx",
"failed",
"unknown",
]);
/** Counts concern retained transaction identities, including those without a request event. */
export const browserScenarioNetworkSummarySchema = z.strictObject({
selected: z.boolean().nullable(),
transactions: count.nullable(),
without_request: count,
ambiguous_transactions: count,
failed_transactions: count,
by_origin: z.array(
z.strictObject({ origin: z.string().nullable(), transactions: count }),
),
by_resource_type: z.array(
z.strictObject({
resource_type: z.string().nullable(),
transactions: count,
}),
),
by_status_class: z.array(
z.strictObject({ status_class: statusClass, transactions: count }),
),
});
interface TransactionGroup {
readonly transaction_id: string;
readonly source_item_indices: readonly number[];
readonly origin: string | null;
readonly resource_type: string | null;
readonly status_class: z.infer<typeof statusClass>;
readonly has_request: boolean;
readonly failed: boolean;
readonly ambiguous: boolean;
}
interface TransactionFacts {
indices: number[];
origins: Set<string | null>;
resourceTypes: Set<string>;
statuses: Set<number>;
requests: number;
failed: boolean;
}
/** Sealed historical captures own this metadata-only index until their retained lifetime ends. */
const transactionsByCapture = new WeakMap<
object,
readonly TransactionGroup[]
>();
/** Keep all observed alternatives; one missing event never erases its transaction. */
export const browserScenarioTransactionGroups = (
capture: RetainedBrowserScenarioCapture,
): readonly TransactionGroup[] => {
const reusable = isImmutableJsonSnapshot(capture);
const known = reusable ? transactionsByCapture.get(capture) : undefined;
if (known !== undefined) return known;
const groups = new Map<string, TransactionFacts>();
for (const [index, event] of capture.events.items.entries()) {
if (!("transaction_id" in event)) continue;
let group = groups.get(event.transaction_id);
if (group === undefined) {
group = {
indices: [],
origins: new Set(),
resourceTypes: new Set(),
statuses: new Set(),
requests: 0,
failed: false,
};
groups.set(event.transaction_id, group);
}
group.indices.push(index);
if (event.kind === "network-content") continue;
group.origins.add(event.url.origin);
group.resourceTypes.add(event.resource_type);
if (event.kind === "request") group.requests += 1;
if (event.kind === "response") group.statuses.add(event.status);
if (event.kind === "request-failed") group.failed = true;
}
const result = [...groups].map(
([transaction_id, group]): TransactionGroup => {
const status = group.statuses.values().next().value;
return Object.freeze({
transaction_id,
source_item_indices: Object.freeze(group.indices),
origin:
group.origins.size === 1
? (group.origins.values().next().value ?? null)
: null,
resource_type:
group.resourceTypes.size === 1
? (group.resourceTypes.values().next().value ?? null)
: null,
status_class:
group.statuses.size === 1 && status !== undefined
? (`${String(Math.floor(status / 100))}xx` as z.infer<
typeof statusClass
>)
: group.statuses.size === 0 && group.failed
? "failed"
: "unknown",
has_request: group.requests > 0,
failed: group.failed,
ambiguous:
group.requests > 1 ||
group.statuses.size > 1 ||
group.origins.size > 1 ||
group.resourceTypes.size > 1,
});
},
);
if (reusable) transactionsByCapture.set(capture, result);
return result;
};
const increment = <Key>(values: Map<Key, number>, key: Key): void => {
values.set(key, (values.get(key) ?? 0) + 1);
};
const sorted = <Key extends string | null>(
values: ReadonlyMap<Key, number>,
): [Key, number][] =>
[...values].sort(([left], [right]) =>
left === right
? 0
: left === null
? -1
: right === null
? 1
: compareAsciiStrings(left, right),
);
/** Null selection/counts preserve legacy unknowns and unrequested network observation. */
export const summarizeBrowserScenarioNetwork = (
capture: RetainedBrowserScenarioCapture,
): z.infer<typeof browserScenarioNetworkSummarySchema> => {
const groups = browserScenarioTransactionGroups(capture);
const contentSelected = Object.values(capture.scenario.network_content).some(
Boolean,
);
const selected =
contentSelected ||
capture.scenario.event_families?.includes("network") === true
? true
: capture.scenario.event_families === undefined
? null
: false;
const origins = new Map<string | null, number>();
const resources = new Map<string | null, number>();
const statuses = new Map<z.infer<typeof statusClass>, number>();
let withoutRequest = 0;
let ambiguous = 0;
let failed = 0;
for (const group of groups) {
if (!group.has_request) withoutRequest += 1;
if (group.ambiguous) ambiguous += 1;
if (group.failed) failed += 1;
increment(origins, group.origin);
increment(resources, group.resource_type);
increment(statuses, group.status_class);
}
return {
selected,
transactions: selected === true || groups.length > 0 ? groups.length : null,
without_request: withoutRequest,
ambiguous_transactions: ambiguous,
failed_transactions: failed,
by_origin: sorted(origins).map(([origin, transactions]) => ({
origin,
transactions,
})),
by_resource_type: sorted(resources).map(
([resource_type, transactions]) => ({ resource_type, transactions }),
),
by_status_class: sorted(statuses).map(([status_class, transactions]) => ({
status_class,
transactions,
})),
};
};
@@ -0,0 +1,388 @@
import type { z } from "zod";
import type { AnalysisError } from "../analysisErrorBase.js";
import {
retainedBrowserScenarioCaptureSchema,
type RetainedBrowserScenarioCapture as BrowserScenarioCapture,
} from "../browserScenarioCapture.js";
import type { BrowserScenarioStep as Step } from "../browserScenarioCaptureValues.js";
import { compareAsciiStrings } from "../canonicalOrdering.js";
import { err, ok, type Result } from "../result.js";
import { analysisViewInputError } from "./analysisViewErrors.js";
import {
completeWithinViewCoverage,
pageViewCoverage,
} from "./analysisViewCoverage.js";
import { createAnalysisViewResultReader } from "./analysisViewParent.js";
import type {
AnalysisViewCoverage,
AnalysisViewParent,
AnalysisViewRequest,
UnsignedAnalysisView,
} from "./analysisView.js";
import {
ARTIFACT_SECTIONS,
stepRowSchema,
browserScenarioSummarySchema,
stepSectionsSchema,
summaryStepSchema,
focusAccessibilitySchema,
browserScenarioViewItemsSchema,
type BrowserScenarioViewRequest,
type ArtifactSection,
} from "./browserScenarioViewSchemas.js";
import {
browserScenarioTransactionGroups,
summarizeBrowserScenarioNetwork,
} from "./browserScenarioNetworkView.js";
const OPERATION = "capture_browser_scenario" as const;
const readCapture = createAnalysisViewResultReader(
retainedBrowserScenarioCaptureSchema,
);
const unsupportedView = (): AnalysisError =>
analysisViewInputError([
{
path: ["view"],
reason: "invalid_value",
message:
'Parent Evidence operation "capture_browser_scenario" supports summary and browser-scenario views, plus compatible environment views.',
},
]);
const sameUrl = (
left: Step["after_url"] | undefined,
right: Step["before_url"],
): boolean =>
left !== undefined &&
left.url === right.url &&
left.redacted === right.redacted;
const stepSections = (step: Step): z.infer<typeof stepSectionsSchema> => {
const sections: z.infer<typeof stepSectionsSchema> = {
captured: [],
missing: [],
truncated: [],
};
for (const section of ARTIFACT_SECTIONS) {
const { state } = step.artifacts[section];
if (state !== "not_requested") sections[state].push(section);
}
return sections;
};
const summaryStep = (
step: Step,
previous: Step | undefined,
): z.infer<typeof summaryStepSchema> => ({
step_index: step.step_index,
step_id: step.step_id,
action: step.action,
status: step.status,
error: step.error,
elapsed_ms: step.elapsed_ms,
...(sameUrl(previous?.after_url, step.before_url)
? {}
: { before_url: step.before_url }),
after_url: step.after_url,
events: {
sequence_start: step.event_sequence_start,
sequence_end: step.event_sequence_end,
},
sections: stepSections(step),
completeness: step.completeness.status,
});
/** The first unsuccessful step explains a failure; otherwise the final screen does. */
const focusAccessibility = (
steps: readonly Step[],
): z.infer<typeof focusAccessibilitySchema> => {
const unsuccessful = steps.findIndex(({ status }) => status !== "completed");
const focusIndex = unsuccessful === -1 ? steps.length - 1 : unsuccessful;
const focus = steps[focusIndex];
if (focus === undefined) throw new Error("Capture has no steps");
const kind =
unsuccessful === -1 ? ("last-step" as const) : "first-unsuccessful-step";
for (let index = focusIndex; index >= 0; index -= 1) {
const accessibility = steps[index]?.artifacts.accessibility;
if (accessibility?.state === "captured")
return {
focus_step_index: focusIndex,
focus: kind,
step_index: index,
accessibility,
};
}
return {
focus_step_index: focusIndex,
focus: kind,
step_index: null,
accessibility: focus.artifacts.accessibility,
};
};
/** Counts in canonical key order; a null origin sorts first. */
const sortedCounts = <Key extends string | null>(
counts: ReadonlyMap<Key, number>,
): [Key, number][] =>
[...counts].sort(([left], [right]) =>
left === right
? 0
: left === null
? -1
: right === null
? 1
: compareAsciiStrings(left, right),
);
const increment = <Key>(counts: Map<Key, number>, key: Key): void => {
counts.set(key, (counts.get(key) ?? 0) + 1);
};
const eventCounts = (
capture: BrowserScenarioCapture,
): z.infer<typeof browserScenarioSummarySchema>["events"] => {
const kinds = new Map<
BrowserScenarioCapture["events"]["items"][number]["kind"],
number
>();
for (const event of capture.events.items) increment(kinds, event.kind);
const { omissions } = capture.events;
return {
retained: capture.events.retained,
dropped: capture.events.dropped,
by_kind: sortedCounts(kinds).map(([kind, count]) => ({ kind, count })),
omissions: {
by_kind: omissions.by_kind,
network_transactions: omissions.network_transactions.length,
network_transaction_range: omissions.network_transaction_range,
},
};
};
const selected = <Value>(
filter: readonly Value[] | undefined,
): ((value: Value) => boolean) => {
if (filter === undefined) return () => true;
const values = new Set(filter);
return (value) => values.has(value);
};
const stepRow = (
{ artifacts: recorded, ...step }: Step,
selection: ReadonlySet<ArtifactSection>,
): z.infer<typeof stepRowSchema> => {
const artifacts: z.infer<typeof stepRowSchema>["artifacts"] = {};
const omitted: z.infer<typeof stepRowSchema>["omitted_artifacts"] = [];
for (const section of ARTIFACT_SECTIONS) {
const artifact = recorded[section];
if (selection.has(section))
Object.assign(artifacts, { [section]: artifact });
else if (artifact.state !== "not_requested")
omitted.push({ section, state: artifact.state });
}
return { ...step, artifacts, omitted_artifacts: omitted };
};
/** Scan matching rows once and materialize only the requested page. */
const selectedPage = <Input, Row>(
source: readonly Input[],
view: { readonly offset: number; readonly limit: number },
matches: (item: Input, index: number) => boolean,
project: (item: Input, index: number) => Row,
): { rows: Row[]; coverage: AnalysisViewCoverage } => {
const rows: Row[] = [];
let total = 0;
for (const [index, item] of source.entries()) {
if (!matches(item, index)) continue;
if (total >= view.offset && rows.length < view.limit)
rows.push(project(item, index));
total += 1;
}
return { rows, coverage: pageViewCoverage(view.offset, rows.length, total) };
};
/** Text offsets are UTF-16 positions in the exact digest-bound retained artifact. */
const textPage = (
capture: BrowserScenarioCapture,
view: Extract<BrowserScenarioViewRequest, { collection: "text" }>,
) => {
const artifact = capture.steps[view.step_index]!.artifacts[view.artifact];
const value = artifact.state === "captured" ? artifact.value : null;
const text = value?.text.slice(view.offset, view.offset + view.limit) ?? null;
return {
items: {
collection: "text" as const,
step_index: view.step_index,
artifact: view.artifact,
state: artifact.state,
reason: "reason" in artifact ? artifact.reason : null,
sha256: value?.sha256 ?? null,
bytes: value?.bytes ?? null,
observed: artifact.state === "truncated" ? artifact.observed : null,
retained: artifact.state === "truncated" ? artifact.retained : null,
text,
},
coverage: pageViewCoverage(
view.offset,
text?.length ?? 0,
value?.text.length ?? 0,
),
};
};
/** One stable page of exact recorded facts; no whole-result filtering arrays. */
const pageRecords = (
capture: BrowserScenarioCapture,
view: BrowserScenarioViewRequest,
): {
readonly items: z.infer<typeof browserScenarioViewItemsSchema>;
readonly coverage: AnalysisViewCoverage;
} => {
if (view.collection === "text") return textPage(capture, view);
const stepSelected = selected(view.step_indices);
if (view.collection === "steps") {
const artifacts = new Set(view.artifacts);
const page = selectedPage(
capture.steps,
view,
({ step_index }) => stepSelected(step_index),
(step) => stepRow(step, artifacts),
);
return {
items: { collection: "steps", rows: page.rows },
coverage: page.coverage,
};
}
const kindSelected = selected(view.event_kinds);
const transactionSelected = selected(view.transaction_ids);
if (view.collection === "transactions") {
const page = selectedPage(
browserScenarioTransactionGroups(capture),
view,
(group) =>
transactionSelected(group.transaction_id) &&
(view.step_indices === undefined ||
group.source_item_indices.some((index) =>
stepSelected(capture.events.items[index]!.step_index),
)),
(group) => {
const events: {
item_index: number;
event: BrowserScenarioCapture["events"]["items"][number];
}[] = [];
const omitted: {
item_index: number;
kind: BrowserScenarioCapture["events"]["items"][number]["kind"];
}[] = [];
for (const item_index of group.source_item_indices) {
const event = capture.events.items[item_index]!;
if (kindSelected(event.kind)) events.push({ item_index, event });
else omitted.push({ item_index, kind: event.kind });
}
return {
transaction_id: group.transaction_id,
source_item_indices: [...group.source_item_indices],
events,
omitted_events: omitted,
};
},
);
return {
items: { collection: "transactions", rows: page.rows },
coverage: page.coverage,
};
}
const page = selectedPage(
capture.events.items,
view,
(event) =>
stepSelected(event.step_index) &&
kindSelected(event.kind) &&
(view.transaction_ids === undefined ||
("transaction_id" in event &&
transactionSelected(event.transaction_id))),
(event, item_index) => ({ item_index, event }),
);
return {
items: { collection: "events", rows: page.rows },
coverage: page.coverage,
};
};
/** Project retained scenario Evidence without relaunching the browser or replaying actions. */
export const projectBrowserScenarioView = (
parent: AnalysisViewParent,
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
if (view.kind !== "summary" && view.kind !== "browser-scenario")
return err(unsupportedView());
const parsed = readCapture(parent);
if (!parsed.ok) return parsed;
const capture = parsed.value;
const shared = {
parent_evidence_id: parent.evidenceId,
parent_operation: OPERATION,
parent_digest: parent.evidenceId.slice(3),
artifact: null,
limitations: [...parent.limitations],
unknowns: [
"Step states record one controlled replay; they do not establish the application's current state.",
"Event step references record receipt attribution; they do not prove an action caused a transaction.",
...(capture.events.dropped > 0
? [
`${String(capture.events.dropped)} events were dropped by capture budgets; counts and pages cover retained events only.`,
]
: []),
],
};
if (view.kind === "summary")
return ok({
kind: "summary",
view,
...shared,
summary: {
browser: capture.browser,
scenario: capture.scenario,
duration_ms: capture.duration_ms,
completeness: capture.completeness,
steps: capture.steps.map((step, index) =>
summaryStep(step, capture.steps[index - 1]),
),
final_accessibility: focusAccessibility(capture.steps),
events: eventCounts(capture),
network: summarizeBrowserScenarioNetwork(capture),
limitation_count: shared.limitations.length,
},
coverage: completeWithinViewCoverage(1, 1),
});
if (
view.collection === "text" &&
capture.steps[view.step_index] === undefined
)
return err(
analysisViewInputError([
{
path: ["view", "step_index"],
reason: "invalid_value",
message: "The selected step is not present in the retained capture.",
},
]),
);
return ok({
kind: "browser-scenario",
view,
...shared,
source_completeness: capture.completeness,
source_events: {
retained: capture.events.retained,
dropped: capture.events.dropped,
},
source_network_omissions: {
transactions: capture.events.omissions.network_transactions.length,
transaction_range: capture.events.omissions.network_transaction_range,
},
...pageRecords(capture, view),
});
};
@@ -0,0 +1,250 @@
import { z } from "zod";
import { digestSchema } from "../digests.js";
import { browserScenarioNetworkSummarySchema } from "./browserScenarioNetworkView.js";
import { sanitizedBrowserUrlSchema } from "../browserObservation.js";
import { retainedBrowserScenarioCaptureSchema } from "../browserScenarioCapture.js";
import {
browserScenarioCompletenessSchema,
browserScenarioEventKindSchema,
browserScenarioEventSchema,
browserStepArtifactsSchema,
} from "../browserScenarioCaptureValues.js";
export const ARTIFACT_SECTIONS = browserStepArtifactsSchema.keyof().options;
const artifactSectionSchema = browserStepArtifactsSchema.keyof();
export type ArtifactSection = z.infer<typeof artifactSectionSchema>;
const countSchema = z.number().int().nonnegative();
/** Section names by recorded state; not_requested sections are omitted. */
export const stepSectionsSchema = z.strictObject({
captured: z.array(artifactSectionSchema),
missing: z.array(artifactSectionSchema),
truncated: z.array(artifactSectionSchema),
});
export const summaryStepSchema = z.strictObject({
step_index: countSchema,
step_id: z.string().min(1),
action: z.string().min(1),
status: z.enum(["completed", "failed", "cancelled"]),
error: z.string().nullable(),
elapsed_ms: countSchema,
before_url: sanitizedBrowserUrlSchema
.exactOptional()
.describe(
"Present only when it differs from the previous step's after_url (always for step 0).",
),
after_url: sanitizedBrowserUrlSchema,
events: z.strictObject({
sequence_start: z.number().int().min(1),
sequence_end: countSchema,
}),
sections: stepSectionsSchema,
completeness: z.enum(["complete", "incomplete", "truncated"]),
});
export const focusAccessibilitySchema = z.strictObject({
focus_step_index: countSchema,
focus: z.enum(["first-unsuccessful-step", "last-step"]),
/** The step whose tree is returned: the focus step, or the nearest earlier step that captured one. */
step_index: countSchema.nullable(),
accessibility: browserStepArtifactsSchema.shape.accessibility,
});
/** Compact capture overview; complete records stay in the retained parent. */
export const browserScenarioSummarySchema = z.strictObject({
browser: retainedBrowserScenarioCaptureSchema.shape.browser,
scenario: retainedBrowserScenarioCaptureSchema.shape.scenario,
duration_ms: countSchema,
completeness: browserScenarioCompletenessSchema,
steps: z.array(summaryStepSchema),
final_accessibility: focusAccessibilitySchema,
events: z.strictObject({
retained: countSchema,
dropped: countSchema,
by_kind: z.array(
z.strictObject({
kind: browserScenarioEventKindSchema,
count: countSchema,
}),
),
omissions: z.strictObject({
by_kind:
retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape
.by_kind,
network_transactions: countSchema,
network_transaction_range:
retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape
.network_transaction_range,
}),
}),
network: browserScenarioNetworkSummarySchema,
limitation_count: countSchema.describe(
"Number of parent Evidence limitations returned with this summary, including inherited provenance caveats.",
),
});
const pageShape = {
kind: z.literal("browser-scenario"),
offset: countSchema.default(0),
limit: z.int().positive().default(64),
};
const stepSelection = {
step_indices: z
.array(countSchema)
.optional()
.describe(
"Exact recorded step indices; omitted selects all, [] selects none.",
),
};
const transactionSelection = {
transaction_ids: z
.array(z.string().min(1))
.optional()
.describe(
"Exact recorded transaction IDs; omitted selects all, [] selects none.",
),
};
/** Collection-specific fields are validated rather than silently ignored. */
export const browserScenarioViewRequestSchema = z.discriminatedUnion(
"collection",
[
z.strictObject({
...pageShape,
...stepSelection,
collection: z.literal("steps"),
artifacts: z
.array(artifactSectionSchema)
.default([])
.describe(
"Return only selected artifact payloads; omitted_artifacts names the other recorded sections.",
),
}),
z.strictObject({
...pageShape,
...stepSelection,
...transactionSelection,
collection: z.literal("events"),
event_kinds: z
.array(browserScenarioEventKindSchema)
.optional()
.describe("Omitted selects all event kinds; [] selects none."),
}),
z.strictObject({
...pageShape,
...stepSelection,
...transactionSelection,
collection: z.literal("transactions"),
event_kinds: z
.array(browserScenarioEventKindSchema)
.default([
"request",
"response",
"request-finished",
"request-failed",
"request-unfinished",
])
.describe(
"Metadata events by default; select network-content explicitly for bodies and header values. [] returns identity and source indices only.",
),
}),
z.strictObject({
...pageShape,
collection: z.literal("text"),
step_index: countSchema,
artifact: z.enum(["accessibility", "dom"]),
limit: z
.int()
.positive()
.default(2048)
.describe("Caller-selected UTF-16 code units, matching offset."),
}),
],
);
export type BrowserScenarioViewRequest = z.output<
typeof browserScenarioViewRequestSchema
>;
/** One exact step whose unselected captured sections are named rather than returned. */
export const stepRowSchema = z.strictObject({
step_index: countSchema,
step_id: z.string().min(1),
action: z.string().min(1),
status: z.enum(["completed", "failed", "cancelled"]),
error: z.string().nullable(),
elapsed_ms: countSchema,
before_url: sanitizedBrowserUrlSchema,
after_url: sanitizedBrowserUrlSchema,
event_sequence_start: z.number().int().min(1),
event_sequence_end: countSchema,
completeness: browserScenarioCompletenessSchema,
artifacts: browserStepArtifactsSchema.partial(),
omitted_artifacts: z.array(
z.strictObject({
section: artifactSectionSchema,
state: z.enum(["captured", "missing", "truncated"]),
}),
),
});
const eventRowSchema = z.strictObject({
item_index: countSchema,
event: browserScenarioEventSchema,
});
export const browserScenarioViewItemsSchema = z.discriminatedUnion(
"collection",
[
z.strictObject({
collection: z.literal("steps"),
rows: z.array(stepRowSchema),
}),
z.strictObject({
collection: z.literal("events"),
rows: z.array(eventRowSchema),
}),
z.strictObject({
collection: z.literal("transactions"),
rows: z.array(
z.strictObject({
transaction_id: z.string().min(1),
source_item_indices: z.array(countSchema),
events: z.array(eventRowSchema),
omitted_events: z.array(
z.strictObject({
item_index: countSchema,
kind: browserScenarioEventKindSchema,
}),
),
}),
),
}),
z.strictObject({
collection: z.literal("text"),
step_index: countSchema,
artifact: z.enum(["accessibility", "dom"]),
state: z.enum(["captured", "missing", "truncated", "not_requested"]),
reason: z.string().nullable(),
sha256: digestSchema.nullable(),
bytes: countSchema.nullable(),
observed: countSchema.nullable(),
retained: countSchema.nullable(),
text: z.string().nullable(),
}),
],
);
/** Capture-wide facts every page repeats so exhaustion is not read as completeness. */
export const browserScenarioViewContextSchema = z.strictObject({
source_completeness: browserScenarioCompletenessSchema,
source_events: z.strictObject({
retained: countSchema,
dropped: countSchema,
}),
source_network_omissions: z.strictObject({
transactions: countSchema,
transaction_range:
retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape
.network_transaction_range,
}),
});
@@ -7,6 +7,7 @@ import {
const symbol = (name: string) => ({
name,
runtime_name: name.slice(1),
address: null,
weak: null,
reexport: null,
@@ -111,6 +112,7 @@ it("preserves ambiguous import facts without adding import fields to exports", (
const imported = {
...symbol("unused"),
name: null,
runtime_name: null,
raw_name: "_value+0x4",
name_status: "ambiguous",
addend: null,
@@ -7,8 +7,6 @@ import {
isNativeInventoryOperation,
nativeInventoryRowsSchemas,
} from "../native/nativeInventory.js";
import { digestSchema } from "../digests.js";
import { providerSchema } from "../evidence.js";
import { err, ok, type Result } from "../result.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import type {
@@ -53,12 +51,6 @@ export const nativeInventoryViewRowSchema = z.strictObject({
record: inventoryRecordSchema.partial().required({ address: true }),
});
export const nativeInventoryViewContext = {
source_provider: providerSchema.nullable(),
source_analysis_profile_digest: digestSchema.nullable(),
source_parameters: jsonValueSchema,
};
export const nativeInventorySummarySchema = z.strictObject({
collection: z.enum(["procedures", "strings", "names"]),
row_count: z.number().int().nonnegative(),
@@ -0,0 +1,301 @@
import { z } from "zod";
import {
nativeLoadImageCheckSchema,
nativeLoadImageObservationSchema,
nativeLoadImageSchema,
nativeUnsupportedLoadImageSchema,
nativeDosLoadImageSchema,
nativeRawLoadImageSchema,
nativeDolLoadImageSchema,
} from "../native/nativeLoadImage.js";
import { nativeAddressObservationSchema } from "../native/nativeAddress.js";
import { jsonValueSchema } from "../jsonValue.js";
import { err, ok, type Result } from "../result.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import type {
AnalysisViewParent,
AnalysisViewRequest,
UnsignedAnalysisView,
} from "./analysisView.js";
import { analysisViewInputError } from "./analysisViewErrors.js";
import { createAnalysisViewResultReader } from "./analysisViewParent.js";
import {
boundedViewMessages,
completeWithinViewCoverage,
pageViewCoverage,
} from "./analysisViewCoverage.js";
const observation = nativeLoadImageObservationSchema.shape;
const rows = {
source_files: observation.source_files.element,
mappings: observation.mappings.element,
relocations: observation.relocations.element,
entry_points: z.strictObject({ address: nativeAddressObservationSchema }),
entry_context: observation.entry_context.element,
checks: nativeLoadImageCheckSchema,
sections: nativeDolLoadImageSchema.shape.sections.element,
};
const defaultFields: Partial<Record<keyof typeof rows, readonly string[]>> = {
relocations: rows.relocations
.pick({
address: true,
segment: true,
offset: true,
status: true,
type: true,
values: true,
})
.keyof().options,
checks: rows.checks
.pick({ name: true, matched: true, file_offset: true, address: true })
.keyof().options,
};
const selection = {
kind: z.literal("native-load-image"),
offset: z.int().nonnegative().default(0),
limit: z.int().positive().default(64),
indices: z.array(z.int().nonnegative()).optional(),
};
const request = <Name extends keyof typeof rows>(name: Name) =>
z.strictObject({
...selection,
collection: z.literal(name),
fields: z
.array(rows[name].keyof())
.optional()
.describe(
name === "relocations"
? "Defaults to address, segment, offset, status, type and values. Select original_bytes_hex or memory_bytes_hex explicitly."
: name === "checks"
? "Defaults to name, matched, file_offset and address. Select expected or observed explicitly; these can contain complete comparison inventories."
: "Omit to return every recorded field in each selected row.",
),
});
/** Pages retain producer order and original indices within the parent Evidence. */
export const nativeLoadImageViewRequestSchema = z.discriminatedUnion(
"collection",
[
request("source_files"),
request("mappings"),
request("relocations"),
request("entry_points"),
request("entry_context"),
request("checks"),
request("sections"),
],
);
const items = <Name extends keyof typeof rows>(name: Name) => {
const records = z.array(
z.strictObject({
source_index: z.int().nonnegative(),
record: rows[name].partial(),
}),
);
return z.strictObject({
collection: z.literal(name),
rows:
name === "checks" || name === "sections" ? records.nullable() : records,
});
};
export const nativeLoadImageViewItemsSchema = z.discriminatedUnion(
"collection",
[
items("source_files"),
items("mappings"),
items("relocations"),
items("entry_points"),
items("entry_context"),
items("checks"),
items("sections"),
],
);
// Keep the format owner's verification states and small identity facts. A
// measured unsupported format must never become a verified empty check set.
export const nativeLoadImageSummarySchema = z.strictObject({
verification: z.union([
nativeUnsupportedLoadImageSchema.omit({
observations: true,
limitations: true,
}),
nativeDosLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
}),
nativeRawLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
}),
nativeDolLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
sections: true,
}),
]),
observation: nativeLoadImageObservationSchema.pick({
executable_format: true,
language_id: true,
compiler_spec_id: true,
image_base: true,
default_address_space: true,
}),
counts: z.strictObject({
source_files: z.int().nonnegative(),
mappings: z.int().nonnegative(),
relocations: z.int().nonnegative(),
entry_points: z.int().nonnegative(),
entry_context: z.int().nonnegative(),
metadata_recovery: z.int().nonnegative().nullable(),
checks: z.int().nonnegative().nullable(),
sections: z.int().nonnegative().nullable(),
}),
});
const readReport = createAnalysisViewResultReader(nativeLoadImageSchema);
type Row = z.output<(typeof rows)[keyof typeof rows]>;
type PageRequest = z.output<typeof nativeLoadImageViewRequestSchema>;
/** Exact source indices need no inventory scan or persistent lookup index. */
const selectRows = (source: readonly (Row | string)[], view: PageRequest) => {
const selectedFields = view.fields ?? defaultFields[view.collection];
const fields =
selectedFields === undefined ? undefined : new Set<string>(selectedFields);
const project = (value: Row | string, source_index: number) => {
const record = typeof value === "string" ? { address: value } : value;
return {
source_index,
record:
fields === undefined
? record
: Object.fromEntries(
Object.entries(record).filter(([key]) => fields.has(key)),
),
};
};
if (view.indices === undefined)
return {
total: source.length,
rows: source
.slice(view.offset, view.offset + view.limit)
.map((value, index) => project(value, view.offset + index)),
};
const indices = [...new Set(view.indices)]
.filter((index) => index < source.length)
.sort((left, right) => left - right);
return {
total: indices.length,
// The format owner validated a dense collection; these indices are in range.
rows: indices
.slice(view.offset, view.offset + view.limit)
.map((index) => project(source[index]!, index)),
};
};
/** Query the owner's parsed historical observations without reopening the target. */
export const projectNativeLoadImageView = (
parent: AnalysisViewParent,
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
if (view.kind !== "summary" && view.kind !== "native-load-image")
return err(
analysisViewInputError([
{
path: ["view"],
reason: "invalid_value",
message:
"Native load-image Evidence supports summary and native-load-image views.",
},
]),
);
const parsed = readReport(parent);
if (!parsed.ok) return parsed;
const report = parsed.value;
const observed = report.observations;
const { observations: _, limitations, ...verification } = report;
const {
checks: __,
sections: ___,
...smallVerification
} = {
checks: undefined,
sections: undefined,
...verification,
};
const {
source_files,
mappings,
relocations,
entry_points,
entry_context,
metadata_recovery,
...identity
} = observed;
const shared = {
parent_evidence_id: parent.evidenceId,
parent_operation: "inspect_native_load_image" as const,
parent_digest: parent.evidenceId.slice(3),
artifact: parent.artifact ?? null,
source_provider: parent.provider ?? null,
source_analysis_profile_digest: parent.analysisProfileDigest ?? null,
source_parameters: jsonValueSchema.parse(parent.parameters ?? null),
limitations: boundedViewMessages(
[...parent.limitations, ...limitations],
parent.evidenceId,
"limitations",
),
unknowns: [
"Counts and pages describe retained provider observations, not complete runtime mappings or relocation behavior.",
"Null collection counts or rows mean not recorded or not applicable, not observed absence.",
...(report.status === "unsupported" ? [report.reason] : []),
],
};
if (view.kind === "summary")
return ok({
...shared,
kind: "summary",
view,
summary: {
verification: smallVerification,
observation: identity,
counts: {
source_files: source_files.length,
mappings: mappings.length,
relocations: relocations.length,
entry_points: entry_points.length,
entry_context: entry_context.length,
metadata_recovery: metadata_recovery?.length ?? null,
checks: "checks" in report ? report.checks.length : null,
sections: "sections" in report ? report.sections.length : null,
},
},
coverage: completeWithinViewCoverage(1, 1),
});
const selected =
view.collection === "checks"
? "checks" in report
? report.checks
: undefined
: view.collection === "sections"
? "sections" in report
? report.sections
: undefined
: observed[view.collection];
const page = selectRows(selected ?? [], view);
return ok({
...shared,
kind: "native-load-image",
verification_status: report.status,
view,
items: nativeLoadImageViewItemsSchema.parse({
collection: view.collection,
rows: selected === undefined ? null : page.rows,
}),
coverage: pageViewCoverage(view.offset, page.rows.length, page.total),
});
};
@@ -0,0 +1,10 @@
import { digestSchema } from "../digests.js";
import { providerSchema } from "../evidence.js";
import { jsonValueSchema } from "../jsonValue.js";
/** Native projections reference the original provider/profile without repeating its full commitment. */
export const nativeViewSourceContext = {
source_provider: providerSchema.nullable(),
source_analysis_profile_digest: digestSchema.nullable(),
source_parameters: jsonValueSchema,
};
+86
View File
@@ -0,0 +1,86 @@
import type { PositionalRead } from "../zipPackageFormat.js";
const UDIF_TRAILER_BYTES = 512;
const UDIF_TRAILER_SIGNATURE = Buffer.from("koly");
const GPT_HEADER_SIGNATURE = Buffer.from("EFI PART");
const GPT_REVISION_1_0 = 0x0001_0000;
const HFS_PLUS_HEADER_END = 1028;
/** Leading bytes covering every raw layout signature below. */
const RAW_SIGNATURE_PREFIX_BYTES = HFS_PLUS_HEADER_END;
/**
* Recognize Apple disk images by structure, not by name or mountability.
*
* UDIF images carry a `koly` trailer. Raw images (UDRW, UDTO and other
* `CRawDiskImage` formats) are plain device bytes, so they are recognized by
* the partition map or the HFS+ volume header that `hdiutil` writes. This
* establishes neither integrity nor that the image will mount. `read` must
* fill each range; a short result means end of file.
*/
export const hasDiskImageSignature = async (
size: number,
read: PositionalRead,
): Promise<boolean> => {
if (!Number.isSafeInteger(size) || size <= 0) return false;
if (size >= UDIF_TRAILER_BYTES) {
const trailer = await read(
size - UDIF_TRAILER_BYTES,
UDIF_TRAILER_SIGNATURE.length,
);
if (Buffer.from(trailer).equals(UDIF_TRAILER_SIGNATURE)) return true;
}
// Read the exact window here so every caller observes the same bytes.
return hasRawDiskImageSignature(
Buffer.from(await read(0, Math.min(size, RAW_SIGNATURE_PREFIX_BYTES))),
);
};
const hasRawDiskImageSignature = (prefix: Buffer): boolean =>
hasGptHeader(prefix) ||
hasApplePartitionMap(prefix) ||
hasMbrPartitionTable(prefix) ||
hasHfsPlusVolumeHeader(prefix);
/** GUID partition table header at LBA 1 of a 512-byte-sector device. */
const hasGptHeader = (prefix: Buffer): boolean => {
if (prefix.length < 528) return false;
if (!prefix.subarray(512, 520).equals(GPT_HEADER_SIGNATURE)) return false;
const headerSize = prefix.readUInt32LE(524);
return (
prefix.readUInt32LE(520) === GPT_REVISION_1_0 &&
headerSize >= 92 &&
headerSize <= 512
);
};
/** Driver descriptor block (`ER`) followed by a partition map entry (`PM`). */
const hasApplePartitionMap = (prefix: Buffer): boolean =>
prefix.length >= 514 &&
prefix[0] === 0x45 &&
prefix[1] === 0x52 &&
prefix[512] === 0x50 &&
prefix[513] === 0x4d;
/** Boot signature plus a well-formed table with at least one used entry. */
const hasMbrPartitionTable = (prefix: Buffer): boolean => {
if (prefix.length < 512 || prefix[510] !== 0x55 || prefix[511] !== 0xaa)
return false;
let used = false;
for (let entry = 446; entry < 510; entry += 16) {
const status = prefix[entry];
if (status !== 0x00 && status !== 0x80) return false;
used ||= prefix[entry + 4] !== 0 && prefix.readUInt32LE(entry + 12) !== 0;
}
return used;
};
/** Unpartitioned HFS+ (`H+`, version 4) or HFSX (`HX`, version 5) volume. */
const hasHfsPlusVolumeHeader = (prefix: Buffer): boolean => {
if (prefix.length < HFS_PLUS_HEADER_END || prefix[1024] !== 0x48)
return false;
const version = prefix.readUInt16BE(1026);
return (
(prefix[1025] === 0x2b && version === 4) ||
(prefix[1025] === 0x58 && version === 5)
);
};
+3
View File
@@ -164,6 +164,9 @@ const redactFailure = (
...(issue.expected === undefined
? {}
: { expected: visit(issue.expected) }),
...(issue.observed === undefined
? {}
: { observed: visit(issue.observed) }),
})),
);
if (error instanceof AnalysisOutputError)
+100 -60
View File
@@ -10,6 +10,10 @@ import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
import { semanticStaticPropertyKey } from "./javascriptAstValues.js";
import { stringValue } from "./javascriptStaticAnalysisHelpers.js";
import {
withoutUnusedFactoryDefaults,
type JavaScriptBundlerFactoryRoles,
} from "./javascriptBundlerFactoryRoles.js";
import {
resolveSemanticBindingState,
type JavaScriptBindingIndex,
@@ -18,6 +22,8 @@ import {
/** Static CommonJS export names. */
export interface StaticExports {
readonly values: readonly string[];
/** Export-shaped writes whose receiver role could not be resolved. */
readonly unknown: number;
}
/** Derive a rename-resistant syntax fingerprint without code execution. */
@@ -32,18 +38,14 @@ export const fingerprintJavaScriptAst = (node: t.Node): string => {
return createHash("sha256").update(JSON.stringify(tokens)).digest("hex");
};
/** Calling-convention roles established by the bundler adapter. */
export interface JavaScriptFactoryExportBindings {
readonly exports: t.Identifier | undefined;
readonly module: t.Identifier | undefined;
readonly loader: t.Identifier | undefined;
}
/** Collect statically declared CommonJS/bundler export names from one factory. */
export const collectJavaScriptExports = (
node: t.Node,
factory?: {
readonly roles: JavaScriptFactoryExportBindings;
readonly roles: Pick<
JavaScriptBundlerFactoryRoles,
"module" | "exports" | "loader" | "unusedDefaults"
>;
readonly index: Pick<
JavaScriptBindingIndex,
"bindings" | "effectsForReferenceWithinActivationAt"
@@ -54,47 +56,61 @@ export const collectJavaScriptExports = (
const index = factory?.index;
const bindings = index?.bindings;
const exports = new Set<string>();
const bindingId = (
reference: t.Node | null | undefined,
): string | undefined =>
bindings !== undefined && t.isIdentifier(reference)
? resolveSemanticBindingState(bindings, reference, reference.name)
?.bindingId
: undefined;
let unknown = 0;
const bindingId = (reference: t.Identifier): string | undefined =>
bindings === undefined
? undefined
: resolveSemanticBindingState(bindings, reference, reference.name)
?.bindingId;
const identity = (
role: keyof JavaScriptFactoryExportBindings,
parameters: readonly (t.Identifier | undefined)[],
fallback: string,
) => {
const id = bindingId(roles?.[role]);
return (reference: t.Node | null): boolean => {
if (!t.isIdentifier(reference)) return false;
if (roles === undefined) return reference.name === fallback;
if (id === undefined || bindings === undefined) return false;
const ids = new Set(
parameters.flatMap((parameter) => {
const id = parameter === undefined ? undefined : bindingId(parameter);
return id === undefined ? [] : [id];
}),
);
return (reference: t.Node | null): ReceiverRole => {
if (!t.isIdentifier(reference)) return undefined;
if (roles === undefined)
return reference.name === fallback ? "role" : undefined;
if (bindings === undefined) return undefined;
const binding = resolveSemanticBindingState(
bindings,
reference,
reference.name,
);
if (binding?.bindingId !== id) return false;
if (binding === undefined || !ids.has(binding.bindingId))
return undefined;
// The calling convention owns the parameter's entry value, not every
// later value assigned to the same lexical binding.
const read = semanticBindingAtRead(
binding,
withoutUnusedFactoryDefaults(binding, roles),
reference,
bindings.parentsByNode,
);
return (
read.status === "ordered" &&
read.binding.initializers.length === 0 &&
if (read.status !== "ordered") return "unresolved";
return read.binding.initializers.length === 0 &&
read.binding.referenceInitializers.length === 0
);
? "role"
: undefined;
};
};
const isExportsAlias = identity("exports", "exports");
const isModuleAlias = identity("module", "module");
const isLoaderAlias = identity("loader", "__webpack_require__");
const receivers = { isExportsAlias, isModuleAlias };
const isExportsAlias = identity(roles?.exports ?? [], "exports");
const isModuleAlias = identity([roles?.module], "module");
const isLoaderAlias = identity([roles?.loader], "__webpack_require__");
const receivers: ExportReceivers = {
isExportsAlias,
isModuleAlias,
unresolved: () => {
unknown += 1;
},
};
traverseJavaScriptAst(node, {
// A default the calling convention never runs declares no exports.
skip: (current) => roles?.unusedDefaults.has(current) === true,
enter: (current) => {
if (t.isAssignmentExpression(current))
collectAssignmentExports(
@@ -111,31 +127,40 @@ export const collectJavaScriptExports = (
"defineProperty",
]);
collectCallExports(current, exports, receivers, intrinsic, (root) => {
if (root === null || !isLoaderAlias(root)) return false;
if (index === undefined) return true;
const loader = isLoaderAlias(root);
if (loader !== "role" || root === null) return loader;
if (index === undefined) return "role";
const facts = index.effectsForReferenceWithinActivationAt(
current,
root,
["d"],
);
return (
facts.memberEffectsIndexed &&
return facts.memberEffectsIndexed &&
!facts.effects.some(
(effect) =>
effect.kind === "member-write" && effect.timing !== "after",
)
);
? "role"
: undefined;
});
}
return undefined;
},
});
return { values: [...exports].sort(compareUnicodeCodePoints) };
return { values: [...exports].sort(compareUnicodeCodePoints), unknown };
};
/**
* `role` holds the parameter's entry value; `unresolved` reads its binding
* where write order is unknown, so the write may or may not be an export.
*/
type ReceiverRole = "role" | "unresolved" | undefined;
interface ExportReceivers {
readonly isExportsAlias: (root: t.Node | null) => boolean;
readonly isModuleAlias: (root: t.Node | null) => boolean;
readonly isExportsAlias: (root: t.Node | null) => ReceiverRole;
readonly isModuleAlias: (root: t.Node | null) => ReceiverRole;
/** Record an export-shaped write whose receiver could not be resolved. */
readonly unresolved: () => void;
}
const semanticTokens = (node: t.Node): string[] => {
@@ -204,19 +229,24 @@ const collectAssignmentExports = (
left: t.LVal | t.OptionalMemberExpression,
right: t.Expression,
output: Set<string>,
{ isExportsAlias, isModuleAlias }: ExportReceivers,
{ isExportsAlias, isModuleAlias, unresolved }: ExportReceivers,
): void => {
const path = memberPath(left);
if (path === null) return;
const root = memberRoot(left);
const exportsAlias = isExportsAlias(root);
// Rebinding the CommonJS alias does not replace module.exports.
if (path.length === 1 && exportsAlias) return;
const exportOffset = exportsAlias
? 1
: isModuleAlias(root) && path[1] === "exports"
? 2
: null;
if (path.length === 1) return;
const exportsAlias = isExportsAlias(root);
const moduleAlias =
exportsAlias === undefined && path[1] === "exports"
? isModuleAlias(root)
: undefined;
if (exportsAlias === "unresolved" || moduleAlias === "unresolved") {
unresolved();
return;
}
const exportOffset =
exportsAlias === "role" ? 1 : moduleAlias === "role" ? 2 : null;
if (exportOffset === null) return;
if (path.length === exportOffset) {
if (t.isObjectExpression(right)) collectObjectKeys(right, output);
@@ -232,27 +262,37 @@ const collectAssignmentExports = (
const collectCallExports = (
call: t.CallExpression,
output: Set<string>,
{ isExportsAlias, isModuleAlias }: ExportReceivers,
{ isExportsAlias, isModuleAlias, unresolved }: ExportReceivers,
intrinsic: boolean,
isLoaderAlias: (root: t.Node | null) => boolean,
isLoaderAlias: (root: t.Node | null) => ReceiverRole,
): void => {
const callee = memberPath(call.callee);
if (callee === null) return;
const definePropertyCall = intrinsic;
const declarationCall =
callee.length === 2 &&
callee[1] === "d" &&
isLoaderAlias(memberRoot(call.callee));
if (!definePropertyCall && !declarationCall) return;
!definePropertyCall && callee.length === 2 && callee[1] === "d"
? isLoaderAlias(memberRoot(call.callee))
: undefined;
if (!definePropertyCall && declarationCall === undefined) return;
const targetNode = call.arguments[0];
const target = memberPath(targetNode);
const exportTarget =
isExportsAlias(targetNode ?? null) ||
(target !== null &&
target.length === 2 &&
isModuleAlias(memberRoot(targetNode)) &&
target[1] === "exports");
if (!exportTarget) return;
const exportsAlias = isExportsAlias(targetNode ?? null);
const moduleAlias =
exportsAlias === undefined &&
target !== null &&
target.length === 2 &&
target[1] === "exports"
? isModuleAlias(memberRoot(targetNode))
: undefined;
if (exportsAlias === undefined && moduleAlias === undefined) return;
if (
exportsAlias === "unresolved" ||
moduleAlias === "unresolved" ||
declarationCall === "unresolved"
) {
unresolved();
return;
}
if (definePropertyCall) {
const name = stringValue(call.arguments[1]);
if (name !== undefined) addExport(output, name);
@@ -1,6 +1,7 @@
import * as t from "@babel/types";
import { javascriptBootstrapAt } from "./javascriptBootstrap.js";
import { javascriptBundlerFactoryRoles } from "./javascriptBundlerFactoryRoles.js";
import { proveJavaScriptBootstrapIntegrity } from "./javascriptBootstrapIntegrity.js";
import {
semanticMemberChain,
@@ -30,6 +31,9 @@ import {
traverseJavaScriptValueAst,
} from "./javascriptSemanticTraversal.js";
/** A standalone Webpack bootstrap uses the module, exports, loader convention. */
const BOOTSTRAP_RUNTIME = "webpack-bootstrap";
/** Loader bindings and literal externals proven for one standalone bootstrap. */
interface BootstrapExternals {
/** The runtime loader and every factory parameter it is passed as. */
@@ -367,14 +371,18 @@ const proveBootstrapExternals = (
});
for (const { factory } of factories) {
// The bootstrap proves `table[key](module, module.exports, loader)`.
const loaderParameter = factory.params[2];
const bindingId = t.isIdentifier(loaderParameter)
? resolveSemanticBindingState(
state,
loaderParameter,
loaderParameter.name,
)?.bindingId
: undefined;
const loaderParameter = javascriptBundlerFactoryRoles(
factory,
BOOTSTRAP_RUNTIME,
).loader;
const bindingId =
loaderParameter === undefined
? undefined
: resolveSemanticBindingState(
state,
loaderParameter,
loaderParameter.name,
)?.bindingId;
if (
bindingId !== undefined &&
singleDefinition(state, bindingId, "parameter")
@@ -416,14 +424,12 @@ const externalSpecifier = (
factory: t.FunctionExpression | t.ArrowFunctionExpression | t.ObjectMethod,
state: JavaScriptSemanticAnalysisState,
): string | undefined => {
const moduleParameter = factory.params[0];
const names = factory.params.map((parameter) =>
t.isIdentifier(parameter) ? parameter.name : null,
);
const moduleParameter = javascriptBundlerFactoryRoles(
factory,
BOOTSTRAP_RUNTIME,
).module;
if (
!t.isIdentifier(moduleParameter) ||
// A duplicated name binds the later argument, not the module object.
new Set(names).size !== names.length ||
moduleParameter === undefined ||
factory.generator ||
factory.async ||
(t.isObjectMethod(factory) && factory.kind !== "method")
@@ -0,0 +1,149 @@
import * as t from "@babel/types";
import {
runtimeFunctionParameters,
semanticStaticPropertyKey,
} from "./javascriptAstValues.js";
import type { JavaScriptSemanticBindingState } from "./javascriptSemanticState.js";
/** Calling-convention roles bound once from one bundler factory's parameters. */
export interface JavaScriptBundlerFactoryRoles {
/** The parameter bound to the module object. */
readonly module: t.Identifier | undefined;
/** Every parameter binding whose entry value is `module.exports`. */
readonly exports: readonly t.Identifier[];
/** The parameter bound to the bundler loader (`__webpack_require__`). */
readonly loader: t.Identifier | undefined;
/**
* Default expressions of role parameters and of `{exports: alias}` targets.
* The convention always supplies the module, its exports object and the
* loader, so these never run and yield no facts.
*/
readonly unusedDefaults: ReadonlySet<t.Node>;
/**
* False when a role parameter uses a pattern whose bindings cannot be
* matched exactly, such as a rest element or a computed or nested key.
* Facts read through the roles are then incomplete, not absent.
*/
readonly exact: boolean;
}
type FactoryRole = "module" | "exports" | "loader";
/** Positional factory roles of a recognized bundler runtime. */
const roleOrder = (runtime: string): readonly FactoryRole[] =>
runtime === "esbuild-commonjs"
? ["exports", "module"]
: runtime.startsWith("esbuild-")
? []
: ["module", "exports", "loader"];
/** Bind a factory's role parameters by position, never by spelling. */
export const javascriptBundlerFactoryRoles = (
factory: t.Function,
runtime: string,
): JavaScriptBundlerFactoryRoles => {
const parameters = runtimeFunctionParameters(factory);
const roles: {
module: t.Identifier | undefined;
exports: t.Identifier[];
loader: t.Identifier | undefined;
} = { module: undefined, exports: [], loader: undefined };
const unusedDefaults = new Set<t.Node>();
let exact = true;
const bindingIdentifier = (node: t.Node): t.Identifier | undefined => {
if (t.isIdentifier(node)) return node;
if (t.isAssignmentPattern(node) && t.isIdentifier(node.left)) {
unusedDefaults.add(node.right);
return node.left;
}
return undefined;
};
for (const [index, role] of roleOrder(runtime).entries()) {
const parameter = parameters[index];
if (parameter === undefined) break;
// A rest element collects this and every later argument into an array.
if (t.isRestElement(parameter)) {
exact = false;
break;
}
// The convention always supplies every role argument, so a role
// parameter's default is dead code whatever its binding pattern.
const pattern = t.isAssignmentPattern(parameter)
? parameter.left
: parameter;
if (t.isAssignmentPattern(parameter)) unusedDefaults.add(parameter.right);
// Loader consumers resolve its calls through the binding's own value
// facts, which cannot yet discount a default the convention never runs.
if (
t.isIdentifier(pattern) &&
(role !== "loader" || pattern === parameter)
) {
// A repeated simple parameter name binds its last argument.
if (boundLater(parameters, index, pattern.name)) continue;
if (role === "exports") roles.exports.push(pattern);
else roles[role] = pattern;
continue;
}
if (role === "module" && t.isObjectPattern(pattern)) {
if (!moduleExportsAliases(pattern, bindingIdentifier, roles.exports))
exact = false;
continue;
}
exact = false;
}
return { ...roles, unusedDefaults, exact };
};
/** `{exports: alias}` binds `module.exports`; other static keys cannot reach it. */
const moduleExportsAliases = (
pattern: t.ObjectPattern,
bindingIdentifier: (node: t.Node) => t.Identifier | undefined,
output: t.Identifier[],
): boolean => {
for (const property of pattern.properties) {
// A shallow rest copy keeps the `exports` property's object.
if (t.isRestElement(property)) return false;
const key = semanticStaticPropertyKey(property.key, property.computed);
if (key === null) return false;
if (key !== "exports") continue;
const alias = bindingIdentifier(property.value);
if (alias === undefined) return false;
output.push(alias);
}
return true;
};
const boundLater = (
parameters: t.Function["params"],
index: number,
name: string,
): boolean =>
parameters
.slice(index + 1)
.some((parameter) =>
Object.hasOwn(t.getBindingIdentifiers(parameter), name),
);
/**
* A role parameter's binding facts at entry. A default the convention never
* runs is not a possible value, so write ordering must not see it either.
*/
export const withoutUnusedFactoryDefaults = (
binding: JavaScriptSemanticBindingState,
roles: Pick<JavaScriptBundlerFactoryRoles, "unusedDefaults">,
): JavaScriptSemanticBindingState => {
const unused = ({
node,
assignment,
}: JavaScriptSemanticBindingState["referenceInitializers"][number]) =>
assignment !== true && roles.unusedDefaults.has(node);
return binding.referenceInitializers.some(unused)
? {
...binding,
referenceInitializers: binding.referenceInitializers.filter(
(initializer) => !unused(initializer),
),
}
: binding;
};
@@ -62,3 +62,215 @@ it.each([
"real",
]);
});
const webpackChunk = (factory: string): string =>
`globalThis.webpackChunkApp.push([[1], {17${factory}}]);`;
/** Run the inert fixture factory under Webpack's calling convention. */
const conventionExports = (factory: string): string[] => {
let table: Record<string, (...values: unknown[]) => void> = {};
runInNewContext(webpackChunk(factory), {
globalThis: {
webpackChunkApp: {
push: ([, modules]: [unknown, typeof table]) => {
table = modules;
},
},
},
});
const module = { exports: {} };
const loader = {
d: (target: object, getters: Record<string, () => unknown>) => {
for (const [name, get] of Object.entries(getters))
Object.defineProperty(target, name, { enumerable: true, get });
},
};
table["17"]?.call(module.exports, module, module.exports, loader);
return Object.getOwnPropertyNames(module.exports).sort();
};
it.each([
"(e, t) { e.exports = {parse: value => value}; }",
"(e, t) { e.exports.member = 1; }",
"(e, t) { e['exports']['computed'] = 1; }",
"(e, t, n) { Object.defineProperty(e.exports, 'defined', {value: 1}); }",
"(e, t, n) { n.d(e.exports, {declared: () => 1}); }",
": (e, t) => { e.exports = {arrow: 1}; }",
": function (e, t) { e.exports = {expression: 1}; }",
"(e, t) { function inner(e) { e.exports = {ghost: 1}; } e.exports.outer = 1; }",
"(e, t) { { let e = {}; e.exports = {ghost: 1}; } }",
"(e, t) { e = {exports: {}}; e.exports.ghost = 1; }",
"(m, module) { module.exports = {ghost: 1}; }",
"() {}",
"(e = {}, t) { e.exports = {defaulted: 1}; }",
"(e = {exports: {}}, t) { e.exports.defaultedMember = 1; }",
"(e = {}, t) { e = {exports: {}}; e.exports.ghost = 1; }",
"(e, t = {}) { t.defaultedExports = 1; }",
"({exports: x}, t) { x.destructured = 1; }",
"({exports}, t) { exports.shorthand = 1; }",
"({exports: x = {}}, t) { x.destructuredDefault = 1; }",
"({'exports': x}, t) { x.stringKey = 1; }",
"({id, exports: x}, t) { x.withOther = 1; }",
"({id}, t) { t.otherKeyOnly = 1; }",
"({exports: x}, t) { x = {}; x.ghost = 1; }",
"({exports: x}, t) { function inner(x) { x.ghost = 1; } x.real = 1; }",
"({exports: x}, t, n) { n.d(x, {declared: () => 1}); Object.defineProperty(x, 'defined', {value: 2}); }",
])("binds factory roles by position for 17%s", (factory) => {
const analysis = analyzeJavaScriptStaticSource(webpackChunk(factory));
expect({
exports: analysis.bundler_registrations[0]?.modules[0]?.exports,
status: analysis.parse_status,
}).toEqual({ exports: conventionExports(factory), status: "complete" });
});
it("ignores unordered replacements of a non-export member write", () => {
const analysis = analyzeJavaScriptStaticSource(
webpackChunk("(e, t) { e.id = 1; for (const k of [1]) { e = {}; } }"),
);
expect(analysis.bundler_registrations[0]?.modules[0]?.exports).toEqual([]);
expect(analysis.parse_status).toBe("complete");
});
it("binds a repeated sloppy parameter name to its last argument", () => {
const factory =
": function (e, t, e) { e.exports = {ghost: 1}; t.second = 1; }";
const analysis = analyzeJavaScriptStaticSource(webpackChunk(factory));
// Module source is strict, so the parser recovers from the name clash.
expect(analysis.analysis_coverage?.syntax.status).toBe("recovered");
expect(analysis.bundler_registrations[0]?.modules[0]).toMatchObject({
exports: conventionExports(factory),
factory_require_name: "e",
});
});
const receiverParameters = [
["(e, t)", "e.exports"],
["(e = {}, t)", "e.exports"],
["(e, t = {})", "t"],
["({exports: x}, t)", "x"],
] as const;
const controlFlowWrites = [
"for (const key of ['x']) { R.forOf = 1; }",
"for (let i = 0; i < 1; i++) { R.forLoop = 1; }",
"for (const key in {a: 1}) { R.forIn = 1; }",
"let i = 0; while (i++ < 1) { R.whileLoop = 1; }",
"do { R.doWhile = 1; } while (false);",
"try { R.tryBlock = 1; } finally {}",
"{ { R.nestedBlock = 1; } }",
"const later = () => { R.arrowClosure = 1; }; later();",
"function later() { R.declaredClosure = 1; } later();",
"if ([1].length > 0) R.conditional = 1;",
];
it.each(
receiverParameters.flatMap(([parameters, receiver]) =>
controlFlowWrites.map((body) => [
parameters,
body.replaceAll("R.", `${receiver}.`),
]),
),
)("keeps the entry receiver of 17%s through %s", (parameters, body) => {
const factory = `${parameters} { ${body} }`;
const analysis = analyzeJavaScriptStaticSource(webpackChunk(factory));
expect({
exports: analysis.bundler_registrations[0]?.modules[0]?.exports,
status: analysis.parse_status,
}).toEqual({ exports: conventionExports(factory), status: "complete" });
});
it.each([
"(...a) { a[0].exports.rest = 1; }",
"(e, ...r) { r[0].restExports = 1; }",
"({...rest}, t) { rest.exports.objectRest = 1; }",
"({exports: {inner}}, t) { t.nested = 1; }",
"({[k]: x}, t) { t.computedKey = 1; }",
"([m], t) { t.arrayPattern = 1; }",
"(e, {a}) { e.exports.exportsPattern = 1; }",
"(e, t, n = {}) { n.d(t, {defaultedLoader: () => 1}); }",
"(e, t, {d}) { d(t, {destructuredLoader: () => 1}); }",
"(e, t) { for (const k of [1]) { e.exports.beforeReplace = 1; e = {exports: {}}; } }",
"(e = {}, t) { for (const k of [1]) { e.exports.beforeReplace = 1; e = {exports: {}}; } }",
"(e, t) { for (const k of [1]) { t.beforeReplace = 1; t = {}; } }",
"(e, t, n) { for (const k of [1]) { n.d(t, {beforeReplace: () => 1}); n = {d() {}}; } }",
"(e, t) { if ([1].length > 0) e = {exports: {}}; e.exports.afterConditional = 1; }",
"(e, t) { const later = () => { e.exports.closure = 1; }; e = {exports: {}}; later(); }",
])("reports unresolved factory role or receiver 17%s as unknown", (factory) => {
const analysis = analyzeJavaScriptStaticSource(webpackChunk(factory));
expect(analysis.bundler_registrations[0]?.modules).toHaveLength(1);
expect(analysis.parse_status).toBe("partial");
expect(analysis.analysis_coverage?.semantics.unknown_findings).toBe(1);
});
/** Exports plus every loader call made under Webpack's calling convention. */
const conventionRun = (factory: string) => {
let table: Record<string, (...values: unknown[]) => void> = {};
runInNewContext(webpackChunk(factory), {
globalThis: {
webpackChunkApp: {
push: ([, modules]: [unknown, typeof table]) => {
table = modules;
},
},
},
});
const loads: string[] = [];
const chunks: string[] = [];
const loader = Object.assign(
(id: unknown) => {
loads.push(String(id));
return {};
},
{
d: (target: object, getters: Record<string, () => unknown>) => {
for (const [name, get] of Object.entries(getters))
Object.defineProperty(target, name, { enumerable: true, get });
},
e: (id: unknown) => {
chunks.push(String(id));
},
},
);
const module = { exports: {} };
table["17"]?.call(module.exports, module, module.exports, loader);
return {
exports: Object.getOwnPropertyNames(module.exports).sort(),
references: [...loads, ...chunks.map((id) => `chunk:${id}`)].sort(),
async_chunk_keys: chunks.sort(),
};
};
it.each([
[
"(e = {}, t = (() => { for (const k of ['x']) { e.exports.ghost = 1; } })()) {}",
"complete",
],
["(e = (t.ghost = 1), t) {}", "complete"],
[
"(e, t = Object.defineProperty(e.exports, 'ghost', {value: 1})) {}",
"complete",
],
["(e = n.d(t, {ghost: () => 1}), t, n) {}", "complete"],
["(e = n(5), t, n) { t.real = 1; }", "complete"],
["(e = n.e(7), t, n) { t.real = 1; }", "complete"],
["(e, t = (n(8), {}), n) { t.real = 1; }", "complete"],
[
"({exports: x} = (n.d(t, {ghost: () => 1}), {exports: {}}), t, n) { x.real = 1; }",
"complete",
],
["({exports: x = (n(6), {ghost: 1})}, t, n) { x.real = 1; }", "complete"],
["(e, t, n, r = n(5)) { t.real = n.e(6); }", "complete"],
["(e, t, n, r = (t.later = 1)) {}", "complete"],
["(e, t, n = (t.ghost = 1, n(9), () => {})) { t.real = 1; }", "partial"],
["(e, {a} = (e.exports.ghost = 1, {})) { e.exports.real = 1; }", "partial"],
])("yields no facts from never-run defaults in 17%s", (factory, status) => {
const analysis = analyzeJavaScriptStaticSource(webpackChunk(factory));
const registration = analysis.bundler_registrations[0];
expect({
exports: registration?.modules[0]?.exports,
references: analysis.references
.map(({ specifier, expression }) => specifier ?? expression)
.sort(),
async_chunk_keys: registration?.async_chunk_keys,
status: analysis.parse_status,
}).toEqual({ ...conventionRun(factory), status });
});
@@ -1,3 +1,4 @@
import * as t from "@babel/types";
import { expect, it } from "vitest";
import { analyzeJavaScriptStaticSource } from "./javascriptStaticAnalysis.js";
@@ -128,3 +129,114 @@ box.trigger;`;
).toEqual(["box.trigger", "box.trigger"]);
expect(effects[0]?.node).not.toBe(effects[1]?.node);
});
const globalFetchWriteSites = (body: string): string[] => {
const source = `var t={},k=[];${body}`;
const parsed = parseJavaScriptSource(source);
if (parsed === null) throw new Error("Expected stored-slot fixture");
const state = completeSemanticSteps(
indexJavaScriptBindingsSteps(parsed.program),
);
return state.ambientWriteEffects
.filter(
({ path }) =>
JSON.stringify(path) === JSON.stringify(["global", "fetch"]),
)
.map(({ node }) =>
node === undefined ? "" : source.slice(node.start ?? 0, node.end ?? 0),
);
};
it.each([
{
name: "an object read from a replaced slot",
body: "var g={};t[k[0]]=g;t[k[1]]=window;g.fetch=replacement;",
sites: [],
},
{
name: "a literal property value whose slot is replaced",
body: "var o={};var h={p:o};h.p=window;o.fetch=replacement;h.p.fetch=other;",
sites: ["h.p.fetch=other"],
},
{
name: "a global read back through an unknown slot",
body: "t[k[0]]=window;t[k[1]].fetch=replacement;",
sites: ["t[k[1]].fetch=replacement"],
},
{
name: "a global read into an alias of an unknown slot",
body: "t[k[0]]=window;var r=t[k[1]];r.fetch=replacement;",
sites: ["r.fetch=replacement"],
},
{
name: "a nested store through an unknown slot",
body: "t[k[0]]={};t[k[1]].x=window;t[k[2]].x.fetch=replacement;",
sites: ["t[k[2]].x.fetch=replacement"],
},
{
name: "a store through an instance prototype",
body: "function F(){}var x=new F();x.__proto__.slot=window;F.prototype.slot.fetch=replacement;",
sites: ["F.prototype.slot.fetch=replacement"],
},
{
name: "a store through an instance constructor",
body: "function F(){}var x=new F();x.constructor.slot=window;F.slot.fetch=replacement;",
sites: ["F.slot.fetch=replacement"],
},
{
name: "a store into a rest copy",
body: "var src={};var {...copy}=src;copy.slot=window;copy.slot.fetch=replacement;",
sites: ["copy.slot.fetch=replacement"],
},
])("resolves stored references for $name", ({ body, sites }) => {
expect(globalFetchWriteSites(body)).toEqual(sites);
});
it("keeps an object read from a replaced slot as a URL base proof", () => {
const worker =
analyzeJavaScriptStaticSource(`import { Worker } from "node:worker_threads";
const t={},k=[];const g={};t[k[0]]=g;t[k[1]]=globalThis;g.URL=replacement;
new Worker(new URL("./worker.js", import.meta.url));`).references.find(
(reference) => reference.kind === "worker",
);
expect(worker?.specifier).toBe("./worker.js");
});
it("indexes unknown-key stores of fresh objects in linear stored-value work", () => {
// Count reads of the stored values' own syntax during ambient indexing only.
const work = (count: number): number => {
const source = `var t={},k=[];${Array.from({ length: count }, (_, i) => `t[k[${String(i)}]]={v:${String(i)}};`).join("")}`;
const parsed = parseJavaScriptSource(source);
if (parsed === null) throw new Error("Expected stored-object fixture");
const index = completeSemanticSteps(
indexJavaScriptBindingsWithReceiversSteps(parsed.program),
);
let counting = false;
let reads = 0;
for (const statement of parsed.program.body) {
if (
!t.isExpressionStatement(statement) ||
!t.isAssignmentExpression(statement.expression)
)
continue;
const value = statement.expression.right;
const type = value.type;
Object.defineProperty(value, "type", {
enumerable: true,
configurable: true,
get: () => {
if (counting) reads += 1;
return type;
},
});
}
counting = true;
completeSemanticSteps(index.indexAmbientWritesSteps());
counting = false;
expect(index.bindings.ambientWriteEffects).toEqual([]);
return reads;
};
// Each stored value is examined a bounded number of times. Re-reading every
// earlier value from every later store grows at least quadratically.
expect(work(64)).toBeLessThan(3 * work(32));
});
@@ -83,6 +83,40 @@ export const retainSemanticAmbientEffect = (
/** Undefined means a cyclic or conflicting projection can select any depth. */
type AmbientProjection = AmbientPath | undefined;
/**
* Select the stored value an object's reference store exposes to a projection,
* or null when the store cannot reach it. A shorter projection only reaches
* stored children when the query covers escaped contents.
*/
const storedValueProjection = (
suffix: AmbientProjection,
source: SemanticIterationSource,
path: AmbientProjection,
escapedContents: boolean,
): AmbientProjection | null => {
if (
suffix !== undefined &&
path !== undefined &&
(!path.every(
(key, index) =>
index >= suffix.length ||
key === null ||
suffix[index] === null ||
key === suffix[index],
) ||
(suffix.length < path.length && !escapedContents))
)
return null;
return suffix === undefined || path === undefined
? undefined
: [
...source.projection.map((key) =>
key === null || typeof key === "object" ? null : String(key),
),
...suffix.slice(path.length),
];
};
const joinAmbientProjection = (
left: AmbientProjection,
right: AmbientProjection,
@@ -129,7 +163,7 @@ export const createAmbientPathResolver = (
capture?: t.Node,
escapedContents = false,
projection: AmbientProjection = [],
localIdentity?: (node: t.Node, path: AmbientProjection) => void,
receiver?: (node: t.Node, path: AmbientProjection) => void,
): AmbientResolution => {
const prior = possibleWrites ? undefined : cached.get(node);
if (prior !== undefined) return prior;
@@ -138,11 +172,20 @@ export const createAmbientPathResolver = (
const unknownReasons = new Set<AmbientIdentityUnknownReason>();
// The caller's write site selects initializers for the written target.
// Reads reached through an initializer occur at their own position.
const pending: {
type Pending = {
node: t.Node;
suffix: AmbientProjection;
readsAtSelf?: true;
}[] = [{ node, suffix: projection }];
};
const pending: Pending[] = [{ node, suffix: projection }];
// A receiver query reports the objects a write to `node` mutates, as
// projections that still end at the written key. Once a step consumes that
// key, the item denotes the value read from the written slot. Replacing
// the slot does not mutate that value or the containers it was read from.
const queue = (item: Pending): void => {
if (receiver !== undefined && item.suffix?.length === 0) return;
pending.push(item);
};
// Join projections at their producing expression, rather than enumerating
// alias histories. Each key can only widen to a wildcard, and conflicting
// depths widen to unknown, so diamonds and cycles have finite state.
@@ -213,7 +256,7 @@ export const createAmbientPathResolver = (
// A widened projection is a possible write, never proof of an intrinsic.
if (suffix === undefined && !possibleWrites) continue;
const continueWith = (node: t.Node, path: AmbientProjection = suffix) =>
pending.push({
queue({
node,
suffix: path,
...(current.readsAtSelf && { readsAtSelf: true }),
@@ -239,50 +282,21 @@ export const createAmbientPathResolver = (
);
};
if (!t.isIdentifier(root) && facts?.freshObject(root)) {
localIdentity?.(root, suffix);
receiver?.(root, suffix);
// The allocation owns itself; a projected child needs its own proof.
if (suffix?.length === 0) {
hasLocalIdentity = true;
resolvedIdentity = true;
}
for (const [source, path] of facts?.stores.get(root) ?? []) {
const selected =
suffix === undefined ||
path === undefined ||
path.every(
(key, index) =>
index >= suffix.length ||
key === null ||
suffix[index] === null ||
key === suffix[index],
);
if (
!selected ||
(suffix !== undefined &&
path !== undefined &&
suffix.length < path.length &&
!escapedContents)
)
continue;
const remaining =
suffix === undefined || path === undefined
? undefined
: suffix.slice(path.length);
pending.push({
node: source.node,
readsAtSelf: true,
suffix:
remaining === undefined
? undefined
: [
...source.projection.map((key) =>
key === null || typeof key === "object"
? null
: String(key),
),
...remaining,
],
});
const stored = storedValueProjection(
suffix,
source,
path,
escapedContents,
);
if (stored === null) continue;
queue({ node: source.node, readsAtSelf: true, suffix: stored });
}
}
if (
@@ -340,7 +354,7 @@ export const createAmbientPathResolver = (
escapedContents ? "escape" : "identity",
);
if (path === null) continue;
pending.push({
queue({
node: initializer.node,
readsAtSelf: true,
suffix:
@@ -667,17 +681,19 @@ export function* indexJavaScriptAmbientWritesSteps(
},
);
const pendingStores = new Set(storedReferences);
const dependents = new WeakMap<t.Node, Set<SemanticIterationSource>>();
// Each dependent keeps the joined projection its latest query reached.
const dependents = new WeakMap<
t.Node,
Map<SemanticIterationSource, AmbientProjection>
>();
while (pendingStores.size > 0) {
const source = pendingStores.values().next().value;
if (source === undefined) break;
pendingStores.delete(source);
if (source.target === undefined) continue;
const identities = new Map<t.Node, AmbientProjection>();
// Record the objects this store mutates, not the values it replaces.
ambientWrites(source.target, source.mutation, false, [], (node, path) => {
const users = dependents.get(node) ?? new Set<SemanticIterationSource>();
users.add(source);
dependents.set(node, users);
identities.set(
node,
identities.has(node)
@@ -685,6 +701,13 @@ export function* indexJavaScriptAmbientWritesSteps(
: path,
);
});
for (const [node, path] of identities) {
const users =
dependents.get(node) ??
new Map<SemanticIterationSource, AmbientProjection>();
users.set(source, path);
dependents.set(node, users);
}
// Publish only completed queries; each source/root projection can only widen.
for (const [node, path] of identities) {
const entries =
@@ -703,8 +726,13 @@ export function* indexJavaScriptAmbientWritesSteps(
continue;
entries.set(source, joined);
stores.set(node, entries);
for (const dependent of dependents.get(node) ?? [])
pendingStores.add(dependent);
// A receiver query drops a value read from its written slot, so only a
// nonempty or widened selection can change a dependent's identities.
for (const [dependent, reached] of dependents.get(node) ?? []) {
const stored = storedValueProjection(reached, source, joined, false);
if (stored !== null && stored?.length !== 0)
pendingStores.add(dependent);
}
}
yield;
}
@@ -1,8 +1,14 @@
import * as t from "@babel/types";
import { expect, it, vi } from "vitest";
import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js";
import {
analyzeJavaScriptSemantics,
analyzeParsedJavaScriptSemantics,
} from "./javascriptSemanticAnalysis.js";
import { onlyBinding } from "./javascriptSemanticAnalysis.fixture.js";
import type { JavaScriptSemanticIr } from "./javascriptSemanticIr.js";
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { parseJavaScriptSource } from "./javascriptSourceParser.js";
/** Count AST ancestry lookups made while analyzing one source. */
const ancestryReads = (
@@ -145,3 +151,97 @@ it.each([
expect(larger).toBeLessThan(8 * smaller);
},
);
/** Count reads of short-circuit left operands while analyzing one source. */
const logicalOperandReads = (
source: string,
inspect: (ir: JavaScriptSemanticIr) => void,
): number => {
const file = parseJavaScriptSource(source);
if (file === null) throw new Error("fixture must parse");
let reads = 0;
traverseJavaScriptAst(file, {
enter: (node) => {
if (!t.isLogicalExpression(node)) return;
const left = node.left;
Object.defineProperty(node, "left", {
enumerable: true,
configurable: true,
get: () => {
reads++;
return left;
},
});
},
});
reads = 0;
inspect(analyzeParsedJavaScriptSemantics(file));
return reads;
};
/** A left-nested chain `o0 op1 o1 op2 o2 ...` as written in tokenizers. */
const logicalChain = (
length: number,
operand: (index: number) => string,
operator: (index: number) => string,
): string =>
Array.from({ length: length - 1 }, (unused, index) => index + 1).reduce(
(chain, index) => `(${chain}) ${operator(index)} ${operand(index)}`,
operand(0),
);
const call = (index: number): string => `f${String(index)}(s)`;
it.each([
["||", call, () => "||"],
["&&", (index: number) => `s.k${String(index)}`, () => "&&"],
["??", call, () => "??"],
[
"mixed ||/&&/?? and conditional",
(index: number) =>
index % 4 === 0
? `(s.k ? ${call(index)} : s.k${String(index)})`
: call(index),
(index: number) => ["||", "&&", "??"][index % 3] ?? "||",
],
])(
"reads each operand of a long %s chain a bounded number of times",
(_, operand, operator) => {
// Logical branch selection evaluates the left operand. Evaluating it again
// as a value branch at every level doubles the work per operand.
const reads = (length: number): number =>
logicalOperandReads(
`function next(s){ var tok=${logicalChain(length, operand, operator)}; return tok; } module.exports=next;`,
(ir) => {
expect(ir.coverage).toEqual({ status: "complete", omittedCount: 0 });
expect(onlyBinding(ir, "tok").value.status).not.toBe("literal");
},
);
const smaller = reads(8);
const larger = reads(16);
expect(smaller).toBeGreaterThan(0);
// Doubling per operand would multiply the reads by 256.
expect(larger).toBeLessThan(4 * smaller);
},
);
it("keeps established logical operands across a long chain", () => {
// Known-falsy operands select the right branch at every level; the value and
// provenance must still come from the one reachable operand.
const ir = analyzeJavaScriptSemantics(
`const tok = ${'false || null || 0 || "" || '.repeat(10)}require("electron").app;`,
);
expect(ir.coverage).toEqual({ status: "complete", omittedCount: 0 });
expect(onlyBinding(ir, "tok").provenance).toEqual({
status: "module",
origins: [
{
specifier: "electron",
importedPath: ["app"],
resolutionMode: "require",
},
],
reason: null,
});
});
@@ -998,25 +998,37 @@ export function* collectSemanticMemberMutationsSteps(
// An active normalized prefix owns its closure before nested traversals.
if (!covering.retain(path, captureContexts)) continue;
// A self alias such as `node = node[key]` revisits its capture with the
// same prefix added on every cycle, so exact paths never repeat. When
// that prefix can never be pruned, escaping it once covers every longer
// effect, including writes below shared children.
// Only a capture already on this alias chain can repeat a prefix.
// same prefix added on every cycle, so exact paths never repeat. A self
// store such as `s.c = s.c.p` adds its segment after the stored key
// instead. When that segment can never be pruned, escaping through it
// once covers every longer effect, including writes below shared
// children. Only a capture already on this alias chain can repeat.
const revisited = current.bindings.has(captureKey);
let repeatedPrefix: PropertyPath | undefined;
for (
let start = 1;
revisited && start <= current.path.length;
start++
) {
const suffix = current.path.slice(start);
if (!current.bindings.has(lifetimeKey(captureKey, suffix))) continue;
repeatedPrefix = current.path.slice(0, start);
break;
}
let repeatedBase = 0;
repeat: for (
let base = 0;
revisited && base < current.path.length;
base++
)
for (let end = base + 1; end <= current.path.length; end++) {
const shorter =
base === 0
? current.path.slice(end)
: [...current.path.slice(0, base), ...current.path.slice(end)];
if (!current.bindings.has(lifetimeKey(captureKey, shorter)))
continue;
repeatedPrefix = current.path.slice(0, end);
repeatedBase = base;
break repeat;
}
if (
repeatedPrefix !== undefined &&
observablePaths.repeatsUnpruned(binding.bindingId, repeatedPrefix)
observablePaths.repeatsUnpruned(
binding.bindingId,
repeatedPrefix.slice(0, repeatedBase),
repeatedPrefix.slice(repeatedBase),
)
) {
markValue(
current.node,
@@ -1974,7 +1986,8 @@ const observableMutationPaths = (
) => PropertyPath | null;
readonly repeatsUnpruned: (
bindingId: string,
prefix: PropertyPath,
base: PropertyPath,
segment: PropertyPath,
) => boolean;
} => {
const ambientPaths = createAmbientPathResolver(state, false);
@@ -2265,22 +2278,28 @@ const observableMutationPaths = (
return path;
};
// A projection's origins depend only on its source origins and key. Once
// repeating a prefix revisits an origin set, every longer repetition stays
// observable and normalization can never prune it. The answer depends only
// on the binding and prefix, so every revisit of a long chain shares it.
// repeating a segment after its base revisits an origin set, every longer
// repetition stays observable and normalization can never prune it. The
// answer depends only on the binding, base and segment, so every revisit of
// a long chain shares it.
const repeated = new Map<string, boolean>();
const repeatsUnpruned = (
bindingId: string,
prefix: PropertyPath,
base: PropertyPath,
segment: PropertyPath,
): boolean => {
const key = JSON.stringify([bindingId, prefix]);
const key = JSON.stringify([bindingId, base, segment]);
const cached = repeated.get(key);
if (cached !== undefined) return cached;
let observer = bindingNodes.get(bindingId);
if (observer !== undefined && base.length > 0) {
observer = project(observer, base);
settle();
}
const signatures = new Set<string>();
let result = false;
while (observer !== undefined && prefix.length > 0) {
observer = project(observer, prefix);
while (observer !== undefined && segment.length > 0) {
observer = project(observer, segment);
settle();
if ((observer.origins?.size ?? 0) === 0) break;
const signature = JSON.stringify(
@@ -1,7 +1,14 @@
import * as t from "@babel/types";
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js";
import {
analyzeJavaScriptSemantics,
analyzeParsedJavaScriptSemantics,
} from "./javascriptSemanticAnalysis.js";
import { onlyCallable } from "./javascriptSemanticAnalysis.fixture.js";
import type { JavaScriptSemanticIr } from "./javascriptSemanticIr.js";
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { parseJavaScriptSource } from "./javascriptSourceParser.js";
const returnedValue = (body: string, expression: string) =>
onlyCallable(
@@ -109,3 +116,88 @@ describe("member-path cycles and captured aliases", () => {
).toBe("unknown");
});
});
describe("member-path cycles through self stores", () => {
/**
* Analyze a source while counting member-object reads. An unclosed cycle
* grows its path on every revisit; failing at a fixed count keeps that
* regression fast instead of exhausting the test worker's heap.
*/
const analyzeWithBoundedMemberReads = (
source: string,
limit: number,
): { readonly ir: JavaScriptSemanticIr; readonly reads: number } => {
const file = parseJavaScriptSource(source);
if (file === null) throw new Error("fixture must parse");
let reads = 0;
traverseJavaScriptAst(file, {
enter: (node) => {
if (!t.isMemberExpression(node)) return;
const object = node.object;
Object.defineProperty(node, "object", {
enumerable: true,
configurable: true,
get: () => {
if (++reads > limit)
throw new Error(`member-object reads exceeded ${String(limit)}`);
return object;
},
});
},
});
reads = 0;
const ir = analyzeParsedJavaScriptSemantics(file);
return { ir, reads };
};
it.each([
["one key", "s.c = s.c.p; s.c.i = 1;"],
["a nested key", "s.a.b = s.a.b.next; s.a.b.x = 1;"],
["a computed key", "s[k] = s[k].p; s[k].i = 1;"],
["a two-step segment", "s.c = s.c.p.q; s.c.i = 1;"],
[
"two alternating slots",
"s.c = s.d.p; s.d = s.c.q; s.c.i = 1; s.d.j = 2;",
],
[
"a CodeMirror-style context stack",
`s.context = new Context(s.context);
s.context = s.context.prev;
s.context.indent = 0;
s.context.line.firstWord = "x";`,
],
])(
"closes a parameter slot that stores a value read from itself through %s",
(_shape, body) => {
// Each call can see the previous call's store, so the written receiver
// may be any number of segments below the slot's original value.
const { ir, reads } = analyzeWithBoundedMemberReads(
`function Context(prev) { this.prev = prev; this.line = {}; }
function update(s, k) { ${body} }
module.exports = update;`,
5_000,
);
expect(ir.coverage).toEqual({ status: "complete", omittedCount: 0 });
expect(reads).toBeGreaterThan(0);
},
);
it("keeps unrelated slots exact when a local stack pops in a loop", () => {
const body = `
const seed = {
context: { prev: { prev: { value: 1 }, value: 1 }, value: 1 },
keep: 3,
};
const state = seed;
while (state.flag) state.context = state.context.prev;
state.context.value = 2;
`;
expect(returnedValue(body, "seed.context.prev.prev.value")?.status).toBe(
"unknown",
);
expect(returnedValue(body, "seed.keep")).toEqual({
status: "literal",
value: 3,
});
});
});
@@ -721,8 +721,10 @@ const evaluateExpressionValue = (
if (t.isTemplateLiteral(node)) return evaluateTemplate(node, context);
if (t.isConditionalExpression(node) || t.isLogicalExpression(node))
return mergeValues(
semanticSelectedBranches(node, context).map((branch) =>
evaluateExpression(branch, nestedContext(context)),
semanticSelectedBranches(node, context).map(
(branch) =>
branch.value ??
evaluateExpression(branch.node, nestedContext(context)),
),
);
if (t.isObjectExpression(node)) return evaluateObject(node, context);
@@ -774,27 +776,39 @@ const evaluateTemplate = (
return primitiveSet(candidates);
};
/**
* A branch that may produce the expression's value. A logical left operand
* carries the value its selection already evaluated, so value consumers reuse
* it instead of re-evaluating that operand; re-evaluating it at every level of
* a left-nested `a || b || c` chain doubles the work per operand.
*/
interface SemanticSelectedBranch {
readonly node: t.Node;
readonly value?: JavaScriptSemanticValue;
}
/** Value and provenance selection consume the same established branch facts. */
const semanticSelectedBranches = (
node: t.ConditionalExpression | t.LogicalExpression,
context: EvaluationContext,
): readonly t.Node[] => {
): readonly SemanticSelectedBranch[] => {
if (t.isConditionalExpression(node)) {
const test = evaluateExpression(node.test, nestedContext(context));
const choice = semanticCondition(test, Boolean);
return choice === undefined
? [node.consequent, node.alternate]
: [choice ? node.consequent : node.alternate];
? [{ node: node.consequent }, { node: node.alternate }]
: [{ node: choice ? node.consequent : node.alternate }];
}
const left = evaluateExpression(node.left, nestedContext(context));
const value = evaluateExpression(node.left, nestedContext(context));
const left = { node: node.left, value };
const choice = logicalRightOperand(
node.operator,
semanticCondition(left, Boolean),
semanticCondition(left, (value) => value === null, false),
semanticCondition(value, Boolean),
semanticCondition(value, (candidate) => candidate === null, false),
);
return choice === undefined
? [node.left, node.right]
: [choice ? node.right : node.left];
? [left, { node: node.right }]
: [choice ? { node: node.right } : left];
};
/** All retained alternatives must agree before discarding a branch. */
@@ -1320,7 +1334,7 @@ const provenanceForExpression = (
}
if (t.isConditionalExpression(node) || t.isLogicalExpression(node)) {
const candidates = semanticSelectedBranches(node, context).map((branch) =>
provenanceForExpression(branch, nestedContext(context)),
provenanceForExpression(branch.node, nestedContext(context)),
);
const origins = uniqueSemanticOrigins(
candidates.flatMap((candidate) => candidate.origins),
@@ -176,6 +176,8 @@ function* traverseStaticSourceSteps(
routeScan: JavaScriptRouteScan,
): Generator<void, void> {
yield* traverseJavaScriptAstSteps(file, {
// Code that never runs under the bundler convention yields no findings.
skip: (node) => findings.accumulator.unusedFactoryDefaults.has(node),
enter: (node, parent) => {
findings.accumulator.visitedNodes += 1;
routeScan.inspect(node, parent);
@@ -10,6 +10,7 @@ import { semanticFactoryLoaderMemberAt } from "./javascriptSemanticRequireOrigin
import { traverseJavaScriptAst } from "./javascriptSemanticTraversal.js";
import { javascriptBundlerFactoryRoles } from "./javascriptBundlerFactoryRoles.js";
import {
collectJavaScriptExports,
fingerprintJavaScriptAst,
@@ -20,7 +21,6 @@ import {
argumentValue,
calleeName,
chunkRuntime,
factoryRequireName,
moduleFactory,
modulePropertyName,
range,
@@ -257,44 +257,29 @@ const recoverBundlerModule = (
if (key.startsWith("[computed@") || key.startsWith("[unknown@"))
accumulator.unknownFindings += 1;
const fingerprint = fingerprintJavaScriptAst(factory);
const parameters = runtimeFunctionParameters(factory);
const esbuild = runtime.startsWith("esbuild-");
const identifierAt = (index: number): t.Identifier | undefined => {
const parameter = parameters[index];
return t.isIdentifier(parameter) ? parameter : undefined;
};
const roles = {
exports:
runtime === "esbuild-commonjs"
? identifierAt(0)
: esbuild
? undefined
: identifierAt(1),
module:
runtime === "esbuild-commonjs"
? identifierAt(1)
: esbuild
? undefined
: identifierAt(0),
loader: esbuild ? undefined : identifierAt(2),
};
const roles = javascriptBundlerFactoryRoles(factory, runtime);
// An inexact role pattern leaves this module's export list incomplete.
if (!roles.exact) accumulator.unknownFindings += 1;
// Registration is inspected before its factories, so the source traversal
// skips these never-run defaults when it reaches them.
for (const unused of roles.unusedDefaults)
accumulator.unusedFactoryDefaults.add(unused);
const exports: string[] = [];
accumulator.pendingBundlerExports.push({ factory, roles, exports });
const requireName = esbuild ? null : factoryRequireName(factory);
const requireParameter = esbuild ? undefined : parameters[2];
const requireBindingId = t.isIdentifier(requireParameter)
? (resolveSemanticBindingState(
bindings,
requireParameter,
requireParameter.name,
)?.bindingId ?? null)
: null;
const loader = roles.loader;
const requireName = loader?.name ?? null;
const requireBindingId =
loader === undefined
? null
: (resolveSemanticBindingState(bindings, loader, loader.name)
?.bindingId ?? null);
if (requireBindingId !== null)
accumulator.factoryRequireBindingIds.add(requireBindingId);
const asyncChunks = collectBundlerAsyncChunkKeys(
factory,
requireBindingId,
bindings,
roles.unusedDefaults,
);
if (typeof factory.start === "number" && typeof factory.end === "number") {
const moduleRange = {
@@ -393,6 +378,7 @@ const collectBundlerAsyncChunkKeys = (
factory: BundlerFunction,
requireBindingId: string | null,
bindings: JavaScriptSemanticAnalysisState,
unusedDefaults: ReadonlySet<t.Node>,
): StaticValues =>
requireBindingId === null
? { values: [], unknown: 0 }
@@ -401,6 +387,7 @@ const collectBundlerAsyncChunkKeys = (
requireBindingId,
bindings,
"e",
unusedDefaults,
);
const collectBundlerCallArgumentValues = (
@@ -408,12 +395,14 @@ const collectBundlerCallArgumentValues = (
bindingId: string | null,
bindings: JavaScriptSemanticAnalysisState,
member: string | null,
unusedDefaults: ReadonlySet<t.Node> = new Set(),
): StaticValues => {
if (bindingId === null) return { values: [], unknown: 0 };
const values: string[] = [];
let unknown = 0;
const bindingIds = new Set([bindingId]);
traverseJavaScriptAst(factory, {
skip: (node) => unusedDefaults.has(node),
enter: (node) => {
if (!t.isCallExpression(node) && !t.isNewExpression(node))
return undefined;
@@ -463,5 +452,7 @@ export const finishBundlerExports = (
index,
});
for (const value of collected.values) pending.exports.push(value);
// An unordered receiver read leaves the export list incomplete.
accumulator.unknownFindings += collected.unknown;
}
};
@@ -13,7 +13,6 @@ import type {
} from "./javascriptStaticAnalysisTypes.js";
import {
readExactJavaScriptLiteral,
runtimeFunctionParameters,
semanticStaticPropertyKey,
semanticStaticPropertyName,
unwrapJavaScriptExpression,
@@ -151,14 +150,6 @@ export const modulePropertyName = (
: `[unknown@${String(property.start ?? -1)}]`;
};
/** Read the factory-local bundler require parameter when declared. */
export const factoryRequireName = (
factory: t.FunctionExpression | t.ArrowFunctionExpression | t.ObjectMethod,
): string | null => {
const parameter = runtimeFunctionParameters(factory)[2];
return t.isIdentifier(parameter) ? parameter.name : null;
};
/** Collect all unique static literal values from an array. */
export const staticArrayValues = (
array: t.ArrayExpression,
@@ -1,5 +1,5 @@
import type * as t from "@babel/types";
import type { JavaScriptFactoryExportBindings } from "./javascriptAstFingerprint.js";
import type { JavaScriptBundlerFactoryRoles } from "./javascriptBundlerFactoryRoles.js";
import type { JavaScriptOpenReceiverFact } from "./javascriptSemanticAnalysis.js";
import type { JavaScriptRpcScan } from "./javascriptRpcEndpoints.js";
import type { JavaScriptStaticPathIntrinsicQuery } from "./javascriptStaticAnalysisHelpers.js";
@@ -44,9 +44,11 @@ export interface LocatedJavaScriptFinding<Value> {
export interface JavaScriptAnalysisAccumulator {
readonly pendingBundlerExports: {
readonly factory: t.Node;
readonly roles: JavaScriptFactoryExportBindings;
readonly roles: JavaScriptBundlerFactoryRoles;
readonly exports: string[];
}[];
/** Factory parameter defaults the bundler calling convention never runs. */
readonly unusedFactoryDefaults: Set<t.Node>;
readonly pendingModuleRequireCalls: (
| t.CallExpression
| t.OptionalCallExpression
@@ -160,6 +162,7 @@ export const createJavaScriptAnalysisAccumulator = (
sourceLength: number,
): JavaScriptAnalysisAccumulator => ({
pendingBundlerExports: [],
unusedFactoryDefaults: new Set(),
pendingModuleRequireCalls: [],
appPathChanges: [],
references: [],
@@ -1,5 +1,6 @@
import { digestCanonicalValue } from "../canonicalDigest.js";
import type { Evidence } from "../evidence.js";
import { machoRuntimeSymbolName } from "../native/machoSymbolName.js";
import type { ManagedNativeObservationFacts } from "./managedNativeVerificationSource.js";
import type { ManagedNativeBoundaryInspection } from "./managedArtifact.js";
import type {
@@ -40,13 +41,14 @@ export const collectNativeSymbols = (
return { symbols, accepted, unsupported, moduleFormatsByEvidenceId };
};
const normalizedSymbolName = (native: NativeSymbol): string =>
native.source === "macho-export" && native.name.startsWith("_")
? native.name.slice(1)
/** P/Invoke resolves a Mach-O export through dlsym, which names it without the symbol-table prefix. */
const lookupSymbolName = (native: NativeSymbol): string | null =>
native.source === "macho-export"
? machoRuntimeSymbolName(native.name)
: native.name;
const sameSymbolName = (expected: string, native: NativeSymbol): boolean =>
expected === normalizedSymbolName(native);
expected === lookupSymbolName(native);
type ModuleCompatibility = "match" | "mismatch" | "unknown";
@@ -351,7 +353,7 @@ const matchPinvoke = (
);
const exact = candidates.find(
(symbol) =>
managed.import_name === normalizedSymbolName(symbol) &&
managed.import_name === lookupSymbolName(symbol) &&
moduleRelation(symbol) === "match",
);
const unknownModule = candidates.find(
+25 -3
View File
@@ -239,7 +239,6 @@ const functionFact = async (
await checkpoint(256 + 8 * link.length);
for (const link of symbols?.links ?? [])
await checkpoint(256 + 8 * link.length);
await checkpoint(256 + 8 * inventory.evidence_id.length);
const links = new Set([inventory.evidence_id]);
for (const link of detail?.links ?? []) links.add(link);
for (const link of symbols?.links ?? []) links.add(link);
@@ -290,13 +289,36 @@ const functionFact = async (
};
};
const textLength = (value: unknown): number =>
typeof value === "string" ? value.length : 0;
/**
* Every row becomes exactly one emitted member: its parsed row and member
* (768 bytes), then the report validation and Evidence normalization copies
* that serialization digests (1280 bytes). Names and addresses are shared
* strings, charged for UTF-16 storage and serialized copies.
*/
const rowBytes = (row: unknown): number =>
typeof row === "object" && row !== null
? 2048 +
4 *
(textLength((row as { address?: unknown }).address) +
textLength((row as { value?: unknown }).value))
: 2048;
/** Consume application-admitted normalized facts; never infer body extents from neighbor entries. */
export const readFunctionInventory = async (
inventory: Evidence,
observations: readonly Evidence[],
checkpoint: NativeMatchingCheckpoint,
): Promise<MatchingFunction[]> => {
const rows = sourcePayload(matchingRowsSchema, inventory.normalized_result);
// Charge each row's member before parsing clones it, so a borrowed retained
// inventory is bounded without walking the resident record and an oversized
// side is refused before allocation. Members share one inventory ID string.
const source = inventory.normalized_result;
if (Array.isArray(source))
for (const row of source as unknown[]) await checkpoint(rowBytes(row));
await checkpoint(256 + 8 * inventory.evidence_id.length);
const rows = sourcePayload(matchingRowsSchema, source);
const facts: Facts = {
details: new Map(),
symbols: new Map(),
@@ -320,7 +342,7 @@ export const readFunctionInventory = async (
const functions: MatchingFunction[] = [];
for (let sourceIndex = 0; sourceIndex < rows.length; sourceIndex++) {
const row = rows[sourceIndex]!;
await checkpoint(1024 + 4 * (row.address.length + row.value.length));
await checkpoint();
functions.push(
await functionFact({ inventory, row, sourceIndex }, facts, checkpoint),
);
+49
View File
@@ -0,0 +1,49 @@
import { expect, it } from "vitest";
import { machoRuntimeSymbolName } from "./machoSymbolName.js";
it("removes exactly the one symbol-table underscore Darwin adds", () => {
expect(
[
"_main",
"__ZN3rea3addEii",
"_$s5swbin6reaAddyS2i_SitF",
"__T0So8NSObjectCMa",
"_OBJC_CLASS_$_ReaThing",
"__OBJC_$_INSTANCE_METHODS_ReaThing",
"_objc_msgSend$addA:b:",
"__mh_execute_header",
"___stack_chk_fail",
].map(machoRuntimeSymbolName),
).toEqual([
"main",
"_ZN3rea3addEii",
"$s5swbin6reaAddyS2i_SitF",
"_T0So8NSObjectCMa",
"OBJC_CLASS_$_ReaThing",
"_OBJC_$_INSTANCE_METHODS_ReaThing",
"objc_msgSend$addA:b:",
"_mh_execute_header",
"__stack_chk_fail",
]);
});
it("keeps names the compiler does not prefix and their exact spelling", () => {
for (const name of [
"-[ReaThing addA:b:]",
"+[ReaThing new]",
"ltmp0",
"radr://5614542",
"main",
" _main",
"Main",
])
expect(machoRuntimeSymbolName(name)).toBe(name);
});
it("reports no runtime name when none is known or none remains", () => {
expect(machoRuntimeSymbolName(null)).toBeNull();
expect(machoRuntimeSymbolName("")).toBeNull();
expect(machoRuntimeSymbolName("_")).toBeNull();
expect(machoRuntimeSymbolName("__")).toBe("_");
});
+22
View File
@@ -0,0 +1,22 @@
/**
* Runtime spelling of a Mach-O symbol-table name.
*
* Darwin compilers prefix every language-level symbol with one underscore in
* the Mach-O symbol table: C `_main`, Itanium C++ `__ZN3rea3addEii`, Swift
* `_$s…`, Objective-C class data `_OBJC_CLASS_$_…`. dyld `dlsym` and LLDB name
* those symbols without that one underscore. Names the compiler does not
* prefix, such as Objective-C method symbols (`-[Class selector]`) and
* assembler-local labels, are already spelled as the runtime names them.
*
* The symbol-table name stays the source fact; this is its derived runtime
* name. Returns null when no name is known or nothing remains after the prefix.
*/
export const machoRuntimeSymbolName = (
symbolTableName: string | null,
): string | null => {
if (symbolTableName === null) return null;
const runtimeName = symbolTableName.startsWith("_")
? symbolTableName.slice(1)
: symbolTableName;
return runtimeName.length === 0 ? null : runtimeName;
};
+1 -1
View File
@@ -19,7 +19,7 @@ const functionBreakpoint = {
name: nativeString
.min(1)
.describe(
"Exact LLDB runtime symbol name, such as `open`, a C++ or mangled Swift symbol, or `-[NSString length]`. Darwin C symbol-table `_main` may be exposed by LLDB as `main`; REA preserves the selected name without translating it.",
"Exact LLDB runtime symbol name, such as `open`, a C++ or mangled Swift symbol, or `-[NSString length]`. On Darwin, pass an inspect_macho export or import `runtime_name` (`main`), not its symbol-table `name` (`_main`); REA preserves the selected name without translating it.",
),
};
+11
View File
@@ -71,8 +71,18 @@ const segmentSchema = z.object({
initial_permissions: permissionSchema,
sections: coverage(sectionSchema),
});
/** Derived from `name` by the shared Mach-O symbol-name owner. */
const runtimeNameSchema = z
.string()
.nullable()
.default(null)
.describe(
"Symbol-table `name` without the one leading underscore Darwin adds to language-level symbols (`_main` is `main`, `__ZN3rea3addEii` is `_ZN3rea3addEii`), as LLDB and dlsym name it and observe_native_calls function breakpoints accept; a name without that underscore, such as `-[Class selector]`, is unchanged. Null when `name` is null, nothing remains, or the Evidence predates this field.",
);
const symbolSchema = z.object({
/** Exact Mach-O symbol-table name, the source fact. */
name: z.string(),
runtime_name: runtimeNameSchema,
address: hexAddress.nullable(),
weak: z.boolean().nullable(),
reexport: z.boolean().nullable(),
@@ -81,6 +91,7 @@ const symbolSchema = z.object({
const importSymbolSchema = z.object({
/** Resolved symbol identity; null when the textual row has multiple parses. */
name: z.string().nullable(),
runtime_name: runtimeNameSchema,
/** Exact import text retained when a chained suffix makes identity ambiguous. */
raw_name: z.string().nullable().default(null),
name_status: z.enum(["exact", "ambiguous"]).nullable().default(null),
+68 -62
View File
@@ -84,74 +84,80 @@ export const nativeLoadImageCheckSchema = z.strictObject({
address: nativeAddressObservationSchema.nullable(),
});
/** Read-only native image verification; verified applies only to declared coverage. */
export const nativeLoadImageSchema = z.union([
z.strictObject({
status: z.literal("unsupported"),
reason: z.string().min(1),
observations: nativeLoadImageObservationSchema,
limitations: z.array(z.string().min(1)),
/** Measured observations when independent format verification is unsupported. */
export const nativeUnsupportedLoadImageSchema = z.strictObject({
status: z.literal("unsupported"),
reason: z.string().min(1),
observations: nativeLoadImageObservationSchema,
limitations: z.array(z.string().min(1)),
});
export const nativeDosLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.enum(["dos-mz", "dos-com"]),
target_sha256: digest,
load_segment: integer.max(0xffff),
header_bytes: integer,
module_bytes: integer,
overlay_bytes: integer,
entry: z.strictObject({
relative_segment: integer.max(0xffff),
offset: integer.max(0xffff),
linear_address: nativeAddressObservationSchema,
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.enum(["dos-mz", "dos-com"]),
target_sha256: digest,
load_segment: integer.max(0xffff),
header_bytes: integer,
module_bytes: integer,
overlay_bytes: integer,
entry: z.strictObject({
relative_segment: integer.max(0xffff),
offset: integer.max(0xffff),
linear_address: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
export const nativeRawLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("raw-binary"),
target_sha256: digest,
architecture: binaryArchitectureSchema,
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
base_address: nativeAddressObservationSchema,
file_bytes: integer.min(1),
entry: z.strictObject({
address: nativeAddressObservationSchema,
file_offset: z.literal(0),
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("raw-binary"),
target_sha256: digest,
architecture: binaryArchitectureSchema,
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
base_address: nativeAddressObservationSchema,
file_bytes: integer.min(1),
entry: z.strictObject({
address: nativeAddressObservationSchema,
file_offset: z.literal(0),
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("gamecube-dol"),
target_sha256: digest,
architecture: z.literal("powerpc"),
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
file_bytes: integer.min(1),
entry: z.strictObject({ address: nativeAddressObservationSchema }),
sections: z.array(
z.strictObject({
kind: z.enum(["text", "data"]),
index: integer,
file_offset: nativeAddressObservationSchema,
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
),
bss: z.strictObject({
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
export const nativeDolLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("gamecube-dol"),
target_sha256: digest,
architecture: z.literal("powerpc"),
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
file_bytes: integer.min(1),
entry: z.strictObject({ address: nativeAddressObservationSchema }),
sections: z.array(
z.strictObject({
kind: z.enum(["text", "data"]),
index: integer,
file_offset: nativeAddressObservationSchema,
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
),
bss: z.strictObject({
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
/** Read-only native image verification; verified applies only to declared coverage. */
export const nativeLoadImageSchema = z.union([
nativeUnsupportedLoadImageSchema,
nativeDosLoadImageSchema,
nativeRawLoadImageSchema,
nativeDolLoadImageSchema,
]);
/** Provider-neutral native load-image result. */
+17 -3
View File
@@ -54,6 +54,7 @@ import {
ghidraFailureOutput,
GHIDRA_JVM_OOME_MARKER,
GHIDRA_PROCESS_DIAGNOSTIC_BYTES,
type GhidraDiagnosticsScope,
} from "./GhidraDiagnostics.js";
import { GhidraLaunchError, type GhidraLaunch } from "./GhidraLauncher.js";
import { GhidraResponseRouter } from "./GhidraResponseRouter.js";
@@ -158,7 +159,7 @@ export class GhidraClient {
#startupAttempt: GhidraStartupAttempt | undefined;
#closePromise: Promise<void> | undefined;
readonly #failure = bindGhidraSessionFailure(
() => this.#diagnostics(),
(scope) => this.#diagnostics(scope),
(value, incomplete) => this.#redactAuthentication(value, incomplete),
);
readonly #wire: GhidraWire;
@@ -1339,7 +1340,17 @@ export class GhidraClient {
});
}
#diagnostics(): Readonly<Record<string, JsonValue>> {
#diagnostics(
requestedScope: GhidraDiagnosticsScope = "session",
): Readonly<Record<string, JsonValue>> {
// Startup errors still need launcher evidence. A request projection must
// have both a completed handshake and its own active output checkpoint.
const scope =
requestedScope === "request" &&
this.#startupSucceeded() &&
this.#activeOutputCheckpoint !== undefined
? "request"
: "session";
const currentSnapshot = this.#process?.snapshot() ?? this.#processSnapshot;
const snapshot: ProviderProcessSnapshot | undefined =
this.#outputDrainIncomplete && currentSnapshot !== undefined
@@ -1350,7 +1361,9 @@ export class GhidraClient {
}
: currentSnapshot;
const checkpoint =
this.#activeOutputCheckpoint ?? this.#startupOutputCheckpoint;
scope === "request"
? this.#activeOutputCheckpoint
: (this.#activeOutputCheckpoint ?? this.#startupOutputCheckpoint);
const scoped =
checkpoint === undefined
? undefined
@@ -1360,6 +1373,7 @@ export class GhidraClient {
? undefined
: ghidraFailureOutput(scoped, this.#token, this.#outputDrainIncomplete);
return createGhidraDiagnostics({
scope,
targetPath: this.#options.targetPath,
targetSha256: this.#options.targetSha256,
...(this.#targetAdmission === undefined
+58
View File
@@ -5,6 +5,64 @@ import {
} from "./GhidraDiagnostics.js";
import { expect, it } from "vitest";
it("keeps request output and provenance without projecting launcher history", () => {
const output = { text: "launcher output", bytes: 15, observedBytes: 15 };
const options = {
targetPath: "./caller-selected/../target",
targetSha256: "a".repeat(64),
transport: "unix-socket" as const,
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
runtimeRoot: "/owned/runtime",
snapshot: { stdout: output, stderr: output, exitCode: null, signal: null },
previous: {},
failure_output: { stdout: { text: "query output" }, complete: false },
};
const request = createGhidraDiagnostics({ ...options, scope: "request" });
expect(request).toMatchObject({
diagnostic_scope: "request",
target_path: options.targetPath,
target_sha256: options.targetSha256,
profile_digest: options.profileDigest,
runtime_root: options.runtimeRoot,
failure_output: options.failure_output,
});
expect(request).not.toHaveProperty("stdout");
expect(request).not.toHaveProperty("stderr");
expect(createGhidraDiagnostics(options)).toMatchObject({
stdout: { text: output.text },
stderr: { text: output.text },
failure_output: options.failure_output,
});
});
it("keeps cached session output owned while projecting historical request coordinates", () => {
const previous = {
target_path: "/selected/target",
ghidra_log_path: "/owned/project/ghidra.log",
stdout: { text: "old launcher output" },
stderr: { text: "old launcher stderr" },
launch_support: [{ stdout: "JDK version" }],
failure_output: { stdout: { text: "old request" }, complete: true },
};
const request = createGhidraDiagnostics({
scope: "request",
targetPath: "/selected/target",
targetSha256: "a".repeat(64),
transport: "unix-socket",
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
previous,
});
expect(request).toEqual({
diagnostic_scope: "request",
target_path: previous.target_path,
ghidra_log_path: previous.ghidra_log_path,
});
expect(previous.stdout.text).toBe("old launcher output");
expect(previous.failure_output.stdout.text).toBe("old request");
});
it("does not reuse historical operation-scoped failure output without a current owner", () => {
const diagnostics = createGhidraDiagnostics({
targetPath: "/tmp/target",
+25 -9
View File
@@ -13,8 +13,12 @@ export const GHIDRA_PROCESS_DIAGNOSTIC_BYTES = 8 * 1024 * 1024;
/** JVM error class located in all launcher output, beyond the retention budget. */
export const GHIDRA_JVM_OOME_MARKER = "java.lang.OutOfMemoryError";
/** A rejected caller input owns request output, not the launcher's history. */
export type GhidraDiagnosticsScope = "session" | "request";
/** Inputs needed to project bounded diagnostics for one Ghidra runtime. */
export interface GhidraDiagnosticsOptions {
readonly scope?: GhidraDiagnosticsScope;
readonly targetPath: string;
readonly targetSha256: string;
readonly targetAdmission?: JsonValue;
@@ -43,9 +47,16 @@ export const createGhidraDiagnostics = (
) {
const history: Record<string, JsonValue> = { ...options.previous };
delete history.failure_output;
if (options.scope === "request") {
delete history.stdout;
delete history.stderr;
delete history.launch_support;
history.diagnostic_scope = "request";
}
return history;
}
return {
...(options.scope === "request" ? { diagnostic_scope: "request" } : {}),
target_path: options.targetPath,
target_sha256: options.targetSha256,
...(options.targetAdmission === undefined
@@ -71,7 +82,8 @@ export const createGhidraDiagnostics = (
ghidra_log_path: options.launch.ghidraLogPath,
script_log_path: options.launch.scriptLogPath,
process_id: options.launch.process.pid ?? null,
...(options.launch.launchSupport === undefined
...(options.scope === "request" ||
options.launch.launchSupport === undefined
? {}
: {
launch_support: options.launch.launchSupport.map((output) => ({
@@ -90,14 +102,18 @@ export const createGhidraDiagnostics = (
stop_requested: options.snapshot.stopRequested === true,
diagnostic_truncated: options.snapshot.diagnosticTruncated === true,
output_incomplete: options.snapshot.outputIncomplete === true,
stdout: ghidraDiagnosticStream(
options.snapshot.stdout,
options.token,
),
stderr: ghidraDiagnosticStream(
options.snapshot.stderr,
options.token,
),
...(options.scope === "request"
? {}
: {
stdout: ghidraDiagnosticStream(
options.snapshot.stdout,
options.token,
),
stderr: ghidraDiagnosticStream(
options.snapshot.stderr,
options.token,
),
}),
}),
...(options.failure_output === undefined
? {}
+2 -6
View File
@@ -99,6 +99,7 @@ import {
} from "./GhidraProviderCapabilities.js";
import {
GHIDRA_STARTUP_DEADLINE_ELAPSED,
isGhidraInputFailureCode,
type GhidraSessionError,
} from "./GhidraSessionError.js";
import type { GhidraSessionInfo } from "./GhidraSessionValues.js";
@@ -1046,12 +1047,7 @@ const projectPrimarySessionError = (
"Retry this search in literal mode or simplify the regex.",
},
);
if (
failure.kind === "remote" &&
["invalid_request", "not_found", "ambiguous"].includes(
failure.remoteCode ?? "",
)
)
if (failure.kind === "remote" && isGhidraInputFailureCode(failure.remoteCode))
return new AnalysisInputError(operation, context, [
{ path: [], reason: "invalid_value", message: failure.message },
]);
+33
View File
@@ -7,6 +7,39 @@ import {
bindGhidraSessionFailure,
} from "./GhidraSessionError.js";
import { GhidraLaunchError } from "./GhidraLauncher.js";
import { createGhidraDiagnostics } from "./GhidraDiagnostics.js";
it.each(["decompile_failed", "future_remote_failure"])(
"retains complete session diagnostics for non-input remote code %s",
(remoteCode) => {
const factory = bindGhidraSessionFailure((scope) =>
createGhidraDiagnostics({
scope,
targetPath: "/selected/target",
targetSha256: "a".repeat(64),
transport: "unix-socket",
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
runtimeRoot: "/owned/runtime",
snapshot: {
stdout: { text: "engine diagnostic", bytes: 17, observedBytes: 17 },
stderr: { text: "", bytes: 0, observedBytes: 0 },
exitCode: null,
signal: null,
},
previous: {},
}),
);
const failure = factory("remote", "Engine failed", undefined, {
remoteCode,
});
expect(failure.diagnostics).toMatchObject({
stdout: { text: "engine diagnostic" },
remote_code: remoteCode,
});
expect(failure.diagnostics).not.toHaveProperty("diagnostic_scope");
},
);
it("reports truncated cyclic causes in serializable diagnostics", () => {
const cause = new Error("cyclic launcher failure");
+16 -3
View File
@@ -6,6 +6,7 @@ import {
import type { JsonValue } from "../domain/jsonValue.js";
import { DarwinProcessOwnershipInspectionError } from "../process/DarwinProcessRunTokenReader.js";
import { boundedUtf8Text } from "../process/boundedUtf8Text.js";
import type { GhidraDiagnosticsScope } from "./GhidraDiagnostics.js";
import { GhidraLaunchSupportError } from "./GhidraJavaLaunch.js";
import {
GhidraLaunchError,
@@ -184,10 +185,16 @@ const createGhidraSessionError = (failure: {
},
);
/** Bind session-failure construction to a live, token-redacted diagnostic view. */
/** Bridge input failures share classification with their domain error projection. */
export const isGhidraInputFailureCode = (code: string | undefined): boolean =>
code === "invalid_request" || code === "not_found" || code === "ambiguous";
/** Bind session-failure construction to its causal diagnostic scope. */
export const bindGhidraSessionFailure =
(
diagnostics: () => Readonly<Record<string, JsonValue>>,
diagnostics: (
scope: GhidraDiagnosticsScope,
) => Readonly<Record<string, JsonValue>>,
redact: (value: string, incomplete?: boolean) => string = (value) => value,
) =>
(
@@ -203,7 +210,13 @@ export const bindGhidraSessionFailure =
kind,
message: redact(message),
diagnostics: {
...diagnostics(),
...diagnostics(
kind === "remote" &&
(isGhidraInputFailureCode(options.remoteCode) ||
options.remoteCode === "invalid_function_name")
? "request"
: "session",
),
...(cause === undefined
? {}
: { failure_cause: describeCause(cause, redact) }),
+48 -26
View File
@@ -244,6 +244,7 @@ export class LldbCallTracer implements NativeCallTracer {
files: new LldbReadbackFiles(),
supervisor: undefined,
pidPath: paths.pid,
resultPath: paths.result,
targetIdentity: undefined,
targetPid: undefined,
processStopped: true,
@@ -433,16 +434,29 @@ export class LldbCallTracer implements NativeCallTracer {
}
if (owner.processStopped && !owner.targetStopped) {
owner.targetPid = await owner.files.readTargetPid(owner.pidPath);
try {
const targetTermination = await ensureTerminated(
// A stopped LLDB may have published its result before interruption.
const published = await readTracerOutput(owner.resultPath, owner.files);
if (
bridgeProvesTargetGone(
published.ok ? published.value : undefined,
owner.targetPid,
owner.targetIdentity,
);
if (targetTermination !== "terminated")
failures.push(new Error(targetTerminationReason(targetTermination)));
else owner.targetStopped = true;
} catch (cause: unknown) {
failures.push(cause);
)
)
owner.targetStopped = true;
else {
try {
const targetTermination = await ensureTerminated(
owner.targetPid,
owner.targetIdentity,
);
if (targetTermination !== "terminated")
failures.push(
new Error(targetTerminationReason(targetTermination)),
);
else owner.targetStopped = true;
} catch (cause: unknown) {
failures.push(cause);
}
}
}
if (failures.length === 0) {
@@ -646,29 +660,22 @@ export class LldbCallTracer implements NativeCallTracer {
retained.processStopped =
exited.cleanupOwner === undefined && exited.cleanupFailure === undefined;
retained.targetStopped = !exited.targetLaunched;
// The bridge publishes its result once, at the end of rea_trace. Once LLDB
// stopped, an interrupted run still owns that result's target facts, while
// only an exited LLDB selects it as the primary outcome.
const readback =
exited.kind === "exited"
exited.kind === "exited" ||
(exited.targetLaunched && retained.processStopped)
? await readTracerOutput(paths.result, retained.files)
: undefined;
const output = readback?.ok ? readback.value : undefined;
// An explicit bridge observation can prove no process was created or that
// the created process exited. A missing PID alone remains ambiguous.
const bridgeObservedExit =
output !== undefined &&
output.status !== "traced" &&
output.target_process_exit_observed === true;
const published = readback?.ok ? readback.value : undefined;
const output = exited.kind === "exited" ? published : undefined;
retained.targetPid = await retained.files.readTargetPid(paths.pid);
let targetTermination: TargetTermination =
exited.targetLaunched && !bridgeObservedExit
const targetTermination: TargetTermination =
exited.targetLaunched &&
!bridgeProvesTargetGone(published, retained.targetPid)
? await ensureTerminated(retained.targetPid, exited.targetIdentity)
: "terminated";
if (
output !== undefined &&
output.status !== "traced" &&
output.target_process_created === false &&
retained.targetPid === undefined
)
targetTermination = "terminated";
retained.targetStopped = targetTermination === "terminated";
if (
retained.processStopped &&
@@ -1136,6 +1143,20 @@ const readTracerOutput = async (
return output.success ? ok(output.data) : err(output.error);
};
/**
* An explicit bridge observation can prove no process was created (with no
* published PID) or that the created process exited. A missing PID alone, an
* unreadable result, or a traced result remains ambiguous.
*/
const bridgeProvesTargetGone = (
output: z.infer<typeof tracerOutputSchema> | undefined,
targetPid: number | undefined,
): boolean =>
output !== undefined &&
output.status !== "traced" &&
(output.target_process_exit_observed === true ||
(output.target_process_created === false && targetPid === undefined));
const capturedOutput = async (
path: string,
observedBytes: number,
@@ -1184,6 +1205,7 @@ interface RetainedLldbResources {
readonly rootLease: AnalysisCleanupLease;
readonly files: LldbReadbackFiles;
readonly pidPath: string;
readonly resultPath: string;
supervisor: ProviderProcessSupervisor | undefined;
targetIdentity: string | undefined;
targetPid: number | undefined;
+14 -2
View File
@@ -10,6 +10,7 @@ import {
type NativeCommandInvocation,
} from "../domain/native/nativeInspection.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { machoRuntimeSymbolName } from "../domain/native/machoSymbolName.js";
import { err, ok, type Result } from "../domain/result.js";
import type { NativeCommandCapture } from "./CommandRunner.js";
import {
@@ -296,7 +297,9 @@ const normalizeMacho = (
);
const imports = missingImportsUuid
? []
: parseDyldImports(sliceOutput(dyldImportsCapture), hasChainedFixups);
: parseDyldImports(sliceOutput(dyldImportsCapture), hasChainedFixups).map(
withRuntimeName,
);
const imageBase =
load.segments.find(
(segment) => segment.file_offset === 0 && (segment.file_size ?? 0) > 0,
@@ -309,7 +312,9 @@ const normalizeMacho = (
imageBase,
new Set(nmExports.map(({ name }) => name)),
);
const exports = uniqueSymbols([...dyldExports, ...nmExports]);
const exports = uniqueSymbols([...dyldExports, ...nmExports]).map(
withRuntimeName,
);
const dwarfdump = byTool("dwarfdump");
const uuid = reconcileMachoUuid(
load.commands,
@@ -489,6 +494,13 @@ const covered = <Value>(
limitations: [...limitations],
});
const withRuntimeName = <Value extends { readonly name: string | null }>(
symbol: Value,
): Value & { readonly runtime_name: string | null } => ({
...symbol,
runtime_name: machoRuntimeSymbolName(symbol.name),
});
const uniqueSymbols = <Value extends { readonly name: string }>(
items: readonly Value[],
): Value[] => {
+42 -7
View File
@@ -13,6 +13,7 @@ import {
execArgvFromCommandLine,
readLaunchExecArgv,
readLaunchNodeOptions,
regexMatchingDeadlineError,
regexMatchingLimitation,
regexWorkerHeapError,
resolveLaunchRegexWorkerHeap,
@@ -458,12 +459,32 @@ it("describes the effective heap and the configurable matching deadline", () =>
{ kind: "unknown" } as const,
]) {
const limitation = regexMatchingLimitation(heap);
expect(limitation).toContain(
for (const fact of [
"5000 ms",
"128 MiB",
"source default",
"regex.matching_deadline_ms",
"regex.worker_heap_request_mib",
"REA_REGEX_MATCHING_DEADLINE_MS",
"REA_REGEX_WORKER_HEAP_REQUEST_MIB",
"update_analysis_settings",
])
expect(limitation).toContain(fact);
// Providers repeat this text per search operation; bounds and recovery
// stay with the errors below instead of every capability entry.
expect(limitation).not.toContain(
regexMatchingDeadlineRecovery(DEFAULT_REGEX_MATCHING_DEADLINE),
);
expect(limitation).toContain(
expect(limitation).not.toContain(
regexWorkerHeapRequestRecovery(DEFAULT_REGEX_WORKER_HEAP_REQUEST),
);
expect(limitation.length).toBeLessThan(700);
if (heap.kind === "effective")
expect(limitation).toContain(`${String(heap.mib)} MiB`);
expect(
projectAnalysisError(regexMatchingDeadlineError("search_procedures"))
.remediation.action,
).toContain(regexMatchingDeadlineRecovery(DEFAULT_REGEX_MATCHING_DEADLINE));
const projected = projectAnalysisError(
regexWorkerHeapError("search_procedures", new Error("heap"), heap),
);
@@ -478,10 +499,16 @@ it("describes the effective heap and the configurable matching deadline", () =>
expect(projected.remediation.action).toContain(
"--max-old-space-size-percentage",
);
expect(projected.remediation.action).toContain(
regexWorkerHeapRequestRecovery(DEFAULT_REGEX_WORKER_HEAP_REQUEST),
);
if (heap.kind === "unknown") {
expect(limitation).toContain("--max-old-space-size-percentage");
expect(limitation).toContain("--max-heap-size");
expect(limitation).toContain("--experimental-config-file");
expect(projected.remediation.action).toContain(
"--max-old-space-size-percentage",
);
expect(projected.remediation.action).toContain(
"--experimental-config-file",
);
expect(projected.remediation.action).toContain("--max-heap-size");
expect(projected.remediation.action).toContain(
"--experimental-default-config-file",
@@ -515,13 +542,21 @@ it("keeps a selected heap request unless a positive Node flag overrides it", ()
DEFAULT_REGEX_MATCHING_DEADLINE,
request,
);
expect(selected).toContain(regexWorkerHeapRequestRecovery(request));
for (const fact of ["256 MiB", "source setting"])
expect(selected).toContain(fact);
expect(selected).not.toContain("--max-old-space-size");
const overridden = regexMatchingLimitation(
effective(64, true),
DEFAULT_REGEX_MATCHING_DEADLINE,
request,
);
expect(overridden).toContain(regexWorkerHeapRequestRecovery(request));
for (const fact of [
"64 MiB",
"256 MiB",
"source setting",
"--max-old-space-size",
])
expect(overridden).toContain(fact);
const projected = projectAnalysisError(
regexWorkerHeapError(
"search_procedures",
+15 -7
View File
@@ -14,6 +14,7 @@ import {
import { err, ok, type Result } from "../domain/result.js";
import {
DEFAULT_REGEX_MATCHING_DEADLINE,
REGEX_MATCHING_DEADLINE_SETTING,
regexMatchingDeadlineRecovery,
type RegexMatchingDeadline,
} from "../config/regexMatchingDeadline.js";
@@ -413,18 +414,25 @@ const heapBound = (
: `${override}. This process has no positive override, so the effective heap is ${String(heap.mib)} MiB. ${recovery}`;
};
/** Capability text for the supervised ECMAScript matching boundary. */
/**
* Capability text for the supervised ECMAScript matching boundary: the
* effective budgets, their sources, and where to change them. Providers repeat
* it per search operation, so setting bounds and step-by-step recovery stay
* with the deadline and heap errors instead.
*/
export const regexMatchingLimitation = (
heap: RegexWorkerHeapLimit,
deadline: RegexMatchingDeadline = DEFAULT_REGEX_MATCHING_DEADLINE,
request: RegexWorkerHeapRequest = DEFAULT_REGEX_WORKER_HEAP_REQUEST,
): string => {
const deadlineText = `${String(deadline.deadlineMs)} ms matching deadline (source ${deadline.source})`;
const preface =
heap.kind === "effective"
? `Regex mode uses ECMAScript Unicode semantics in a cancellable worker with a ${deadlineText} and an effective ${String(heap.mib)} MiB old-generation heap.`
: `Regex mode uses ECMAScript Unicode semantics in a cancellable worker with a ${deadlineText}.`;
return `${preface} ${regexMatchingDeadlineRecovery(deadline)} ${heapBound(heap, request)} Matching failure leaves the service available.`;
const requested = `${String(request.heapMiB)} MiB worker heap request (source ${request.source})`;
const heapText =
heap.kind === "unknown"
? `a ${requested}; launch heap flags can change the effective old-generation heap, which was not established`
: heap.overridden
? `an effective ${String(heap.mib)} MiB old-generation heap set by a positive Node --max-old-space-size over its ${requested}`
: `an effective ${String(heap.mib)} MiB old-generation heap from its ${requested}`;
return `Regex mode uses ECMAScript Unicode semantics in a cancellable worker with a ${String(deadline.deadlineMs)} ms matching deadline (source ${deadline.source}) and ${heapText}. inspect_analysis_settings and update_analysis_settings read and change regex.matching_deadline_ms and regex.worker_heap_request_mib; ${REGEX_MATCHING_DEADLINE_SETTING} and ${REGEX_WORKER_HEAP_REQUEST_SETTING} seed them at launch. Deadline and heap errors report the bounds and recovery. Matching failure leaves the service available.`;
};
/** Capability text for this process's launch heap, deadline, and heap request. */
Loaded 100 of 146 files, more files were not shown because too many files have changed in this diff. Show more