diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index f87a4e1103..7e35c0db04 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -2,7 +2,7 @@ name: reverse-engineer-anything description: Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. metadata: - version: "36" + version: "37" --- # REA @@ -25,57 +25,18 @@ a skill installed from repository main may describe capabilities absent from an older npm release. Keep complete inline Evidence when that server does not advertise retained references. -When tools are absent or registration is stale: +If tools are absent, a call is blocked by the client, or a required provider is +unavailable, read [connection and recovery](references/connection-and-recovery.md). +Diagnose the observed blocker, continue useful supported work, and resume the +original task after repair. Client approval, shell PATH, registration and engine +availability have different remedies; avoid repeating setup for all of them. -1. Diagnose without changing files. For Codex, run - `npx -y rea-agents@latest doctor --client codex --json`. Substitute the current - supported client: `claude_code`, `claude_desktop`, `codex`, `cursor`, - `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, `copilot_cli`, - `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`, `hermes`, or `grok_bot`. `grok_bot` has no local - file registration: doctor reports the Grok Bot chat step, and setup does not - write `mcp.json`. If the client is unknown, use `doctor --json` and - inspect its registration results before choosing a setup scope. -2. Distinguish the reason. Missing, malformed, or stale registration needs a - scoped configuration repair. An aligned registration with no tools in the - active session needs a restart/reconnection; doctor cannot prove that the - current agent has connected. A missing provider affects only tasks requiring - that provider: static JavaScript inspection needs neither Hopper nor Ghidra, - and Android inspection has separate bring-your-own JADX/Java prerequisites. -3. For a configuration repair, prepare the read-only plan: - `npx -y rea-agents@latest setup --client codex --dry-run --json`. - Use the current client's ID, show its exact proposed paths, backups, and - changes, and obtain approval before setup writes configuration or installs - Hopper. Setup normally installs the matching bundled skill too; include that - replacement in the reviewed plan. After approval, apply the same scope with - `npx -y rea-agents@latest setup --client codex --yes`. Add `--install-hopper` - only if that separate installation was needed and explicitly approved. -4. Restart/reconnect the affected agent. Verify that REA tools actually appear - in the session, then resume the original investigation. If they remain - absent, inspect the client's MCP launch error rather than repeating setup. - -REA setup never installs or upgrades Node.js, npm, Homebrew, Java, Ghidra, IDA, -JADX, Binwalk, or Unblob. Use existing prerequisites; do not install unrelated -software to repair -MCP registration. For an unsupported client, use manual stdio registration -with a version-pinned `rea-agents` package or continue through the CLI. - -A concrete CLI fallback for an operator-supplied JavaScript tree or ASAR is: - -```bash -npx -y rea-agents@latest analyze-javascript-application /absolute/path/to/app --json > app-evidence.json -jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json -npx -y rea-agents@latest inspect-analysis-view app-view.json -``` - -No MCP registration or native engine is required. The CLI returns the complete -Evidence record directly, and a real application can produce hundreds of -megabytes, so save it to a file rather than reading it into context. Start from -the summary view, then request a module page -(`{"kind": "page", "collection": "modules", "offset": 0, "limit": 32}`) or one -module the same way, or pass the saved Evidence inline to -`trace-application-feature`. Read graph, limitations, and unknowns with the -same care as an MCP result. This fallback does not establish that MCP is -configured. Native CLI tasks still require their selected engine. +Prefer an existing `rea` CLI. When none is installed and REA package acquisition +is within the authorized scope, use `npx -y rea-agents@latest` in its place. +Configuration repair starts with a read-only setup plan; show paths, backups and +skill replacement before applying authorized changes. Preserve existing approval +and target-execution boundaries. REA setup never installs or upgrades unrelated +software, including Node.js, npm, Homebrew, Java, Ghidra, IDA, JADX, Binwalk or Unblob. ## Route the target first @@ -140,9 +101,11 @@ after opening; IDA does not supply `binary_overview`. Consult session availabili when composing broader workflows. Modern direct callers and unsupported dossier facets remain unknown; live IDA results are not replayed from snapshots. -If the app is missing, ask which app to inspect. Resolve a human-readable app -name to one clear installed artifact when possible; ask only when matches are -ambiguous. Never choose an example app on the user's behalf. +If the user says only "my app" without a name, path, or target already selected +for this task, ask which app before searching for artifacts. Paths in unrelated +saved reports and nearby test fixtures do not select the user's target. Resolve +an actual human-readable app name to one clear installed artifact when possible; +ask when matches are ambiguous. Never choose an example app on the user's behalf. In a target-free session, use `open_binary` to bind any archive/package or native target whose analysis tool operates on the active target. Do not call an @@ -157,7 +120,11 @@ For a JavaScript application or ELF layout whose size is unknown or large, call `analyze_javascript_application` or `inspect_binary_layout` with `"detail": "summary"`, then inspect the retained `parent_evidence_id` with `inspect_analysis_view`. -Do not repeat an identical tool call. Make a focused follow-up only when the +Page module identities and select the returned `node_id` before requesting an +item; a single file path can represent several graph roles. Do not repeat an +identical analysis or rejected call without a changed input or resolved blocker. +Recheck session state when a lifecycle change makes that status useful. +Make a focused follow-up only when the returned result leaves a specific question unanswered. If MCP reports `resource_constraint` with `details.resource: "transport"`, call @@ -168,6 +135,14 @@ session to a selected path. The analysis remains complete in the ledger; avoid repeating it merely to request the same oversized response. Complete CLI JSON output also streams without a single MCP frame. +A client can also truncate a successful tool result without a REA transport +error. Treat that preview as incomplete; use the retained parent ID for smaller +views or export when the full record is needed. Do not parse the preview as +canonical JSON or repeat the producer. For JavaScript, read the +[module-view and trace recipes](references/javascript-applications.md) before +drilling into a large graph; a broad bidirectional trace can expand unrelated +containment relationships even after summary analysis. + Every conclusion must distinguish observations, inferences, and unknowns. Cite Evidence IDs, preserve limitations and incomplete coverage, and never imply that static analysis observed execution. Runtime requests execute the declared @@ -175,7 +150,9 @@ target and lifecycle; do not broaden the target or action beyond those fields. Within the user's requested investigation, call available analysis tools directly. REA does not require permission grants or per-call approval flags. -Follow the declared request scope and the host's actual access requirements. +The agent client can independently require approval. Follow the declared request +scope and the host's actual access requirements; use the recovery guide for an +observed client-policy rejection. ## Plan broader investigations diff --git a/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md new file mode 100644 index 0000000000..5e5f6251da --- /dev/null +++ b/.agents/skills/reverse-engineer-anything/references/connection-and-recovery.md @@ -0,0 +1,127 @@ +# Connection and task recovery + +Read this guide only for an observed connection, command, approval or provider +blocker. Keep the user's target and question in view. Explain what prevented the +operation and the next useful step; after repair, continue the investigation. + +Commands below use an installed `rea`. If none is available, use the package +invocation in `SKILL.md` when acquiring REA is authorized. Follow the connected +server's schemas and the installed CLI's help rather than assuming repository +main and an older release expose the same arguments. + +## Choose the remedy from the observed failure + +| Observation | Meaning and next step | +| --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| No REA tools in the active session | Inspect registration with `rea doctor --client codex --json`. A skill installation alone does not register MCP. | +| Missing, malformed or stale registration | Prepare a scoped setup plan; apply only authorized changes with backups. | +| Aligned registration, but no connected tools | Restart/reconnect the client. Doctor checks registration; it cannot establish the current agent's connection. Inspect the client's launch error if reconnection fails. | +| `MCP tool call requires approval, but approval policy is never` | Codex rejected the call before REA executed it. Do not install an engine or repeat setup to solve this client-policy decision. See the approval section below. | +| REA returns `provider_unavailable` or an unavailable tool entry | Read that operation's provider diagnostics or `binary_session.result.tool_availability`. Resolve only the prerequisite needed for the task; a complete tool catalog does not imply every operation is available. | +| `rea` is missing in a child shell, or Node is incompatible | Inspect the actual shell PATH and launcher/runtime. A prefix installation may be absent from a login shell's PATH; this does not establish a broken MCP registration. | + +If the client cannot launch its shell/sandbox helper or read the installed skill, +inspect its launch diagnostic. For example, Codex can refuse helper aliases when +CODEX_HOME is under `/tmp`; use an authorized profile outside the OS temporary +directory. REA setup cannot repair the client's executor. Preserve the sandbox +instead of silently disabling it to make an analysis command work. + +Static JavaScript and ordinary ZIP inventory need neither Hopper nor Ghidra. +HAR inspection is offline and does not require a live browser. Native +decompilation, Android analysis and firmware extraction have their own +bring-your-own prerequisites. Use supported inventory/static evidence when it +answers part of the question, and name the unavailable evidence separately. +Do not substitute another target. Label observations from ordinary shell tools +as such; only actual REA output supplies REA Evidence. + +## Repair registration when needed + +For Codex: + +```bash +rea doctor --client codex --json +rea setup --client codex --dry-run --json +``` + +Use the current supported client's ID: `claude_code`, `claude_desktop`, `codex`, +`cursor`, `gemini_cli`, `windsurf`, `devin`, `opencode`, `antigravity`, +`copilot_cli`, `commandcode`, `qwen_code`, `vscode`, `grok_build`, `omp`, `pi`, +`hermes`, or `grok_bot`. If unknown, inspect `doctor --json` before choosing a +scope. `grok_bot` uses a chat registration step rather than local `mcp.json`. +For unsupported clients, use version-pinned manual stdio registration or CLI. + +Show the plan's exact paths, backups, changes and bundled-skill replacement. +Obtain approval for writes when that concrete scope is not already authorized, +then apply the same scope with `rea setup --client codex --yes`. Add +`--install-hopper` only when separately needed and explicitly approved. Do not +install or upgrade unrelated prerequisites. Restart/reconnect, verify tools +actually appear, and resume the original task. If they remain absent, inspect +the launch failure rather than looping through setup. + +## Codex tool approval is separate from REA readiness + +REA tools have no permission-grant handshake. Codex can still require approval +because an operation creates owned resources, launches a helper or has cleanup +effects. An advertised tool or successful doctor result does not override that +client policy. `workspace-write` alone does not resolve an MCP approval +rejection, and `approval_policy = "never"` does not mean every MCP tool is +preapproved. + +When interactive approval is available, use the client's approval flow for the +requested operation. In an unattended run, continue through an authorized CLI +workflow if the execution environment permits it, or report the blocked +operation and the approval needed to proceed. Do not retry the same rejected +call, silently broaden approvals, or rewrite MCP effects as read-only. + +For an operator who explicitly wants to preapprove named operations in a +disposable Codex profile, current Codex supports per-tool configuration. For +example, opening and closing an analysis session can be scoped independently: + +```toml +[mcp_servers.rea.tools.open_binary] +approval_mode = "approve" + +[mcp_servers.rea.tools.close_binary] +approval_mode = "approve" +``` + +Use the actual server ID and only the operations whose effects were approved; +this example does not approve inspection, extraction or runtime execution. +Show the proposed config edit and preserve unrelated entries and backups before +writing. Managed policy can constrain client settings. Consult the +[Codex configuration reference](https://learn.chatgpt.com/docs/config-file/config-reference) +for the installed client's supported settings. This is an optional recovery +choice, not a prerequisite for every REA task. + +## Continue through an installed CLI + +Check the command environment only if the launcher fails: + +```bash +command -v rea +command -v node +node --version +rea --version +``` + +Use the public launcher at its known installation path with a supported existing +Node on PATH when the child shell loses a prefix installation. Avoid internal +`dist/cli.js` entrypoints or global toolchain changes. MCP may have its own valid +launcher even when a shell cannot resolve `rea`. + +For a user-supplied JavaScript tree or ASAR, select output files in the authorized +workspace and save complete Evidence before reading a view: + +```bash +rea analyze-javascript-application /absolute/path/to/app --json > app-evidence.json +jq -c '{source: {kind: "inline", evidence: .}, view: {kind: "summary"}}' app-evidence.json > app-view.json +rea inspect-analysis-view app-view.json +``` + +These commands do not register MCP or install an analysis engine. CLI JavaScript +JSON is complete and can be very large; redirect it rather than loading it into +context. Use an existing JSON processor if `jq` is absent, without installing +one just for this recipe. Inspect a module page or item using the same inline +source and the [JavaScript recipes](javascript-applications.md). Preserve graph +coverage, limitations and unknowns. Native CLI analysis still requires its +selected engine, and all CLI operations retain the host's execution permissions. diff --git a/.agents/skills/reverse-engineer-anything/references/javascript-applications.md b/.agents/skills/reverse-engineer-anything/references/javascript-applications.md index 0ee112a511..01b49a8887 100644 --- a/.agents/skills/reverse-engineer-anything/references/javascript-applications.md +++ b/.agents/skills/reverse-engineer-anything/references/javascript-applications.md @@ -1,20 +1,90 @@ # JavaScript and Electron application artifacts Use `analyze_javascript_application` directly on the operator-supplied ASAR or -extracted tree. The complete result, graph, and Evidence context are returned -inline when they fit the MCP frame. Source-map contents are part of the static analysis. Start from its -findings, coverage, unknowns, and graph context, then make a focused follow-up -only when a specific question remains unanswered. +extracted tree. When size is unknown or large and the connected schema advertises +`detail`, request `detail: "summary"`. This returns an `inspect_analysis_view` +Evidence record with a distinct ID and a `normalized_result.parent_evidence_id` +for the complete retained application analysis. Source-map contents are part of +that static analysis. Read findings, coverage, limitations and unknowns, then +follow a specific unanswered question. Older servers without `detail` return +the complete application Evidence inline when it fits. + +## Read a module without repeating analysis + +Use the actual returned parent ID, on the same connection, as `source.evidence_id`. +The following IDs are placeholders to replace with returned values. When the +module's node ID is not already known, page identities before selecting an item. +Pages return only node IDs, paths and roles; choose a useful page size and follow +`coverage.next_offset` when another page matters: + +```json +{ + "source": { + "kind": "retained-evidence", + "evidence_id": "ev_ACTUAL_PARENT_ID" + }, + "view": { "kind": "page", "collection": "modules", "offset": 0, "limit": 16 } +} +``` + +Select the relevant page entry by its exact `node_id` and recorded role: + +```json +{ + "source": { + "kind": "retained-evidence", + "evidence_id": "ev_ACTUAL_PARENT_ID" + }, + "view": { + "kind": "item", + "collection": "modules", + "selector": { "node_id": "jag_node_ACTUAL_NODE_ID" } + } +} +``` + +Module pages include asset, bundled and source module roles, so one path can +identify several entries. Use `selector.path` only when uniqueness is established; +the returned node ID avoids needing a complete inventory to prove uniqueness. On ambiguity, +select the matching node ID from the page or the error's candidates. Follow +`next_offset` when another page matters. An item exposes that module node's +recorded observations, identity, hashes, exports and source ranges; it does not +include every child syntax observation, related edge or original source file. +An absent item fact is not evidence of absence from the whole application. + +## Trace only the remaining question + +Use `trace_application_feature` on the parent application Evidence for one +literal node ID, route, string, API, IPC channel, module, or native export. +Select `incoming` for relationships leading to the seed, `outgoing` for those +leaving it, and `both` when the question needs both. Trace traversal includes +containment relationships, so even a narrow seed can expand many nodes; a +bidirectional trace can reach unrelated sibling modules. A module item or page +is preferable when its observations already answer the question. Do not treat +all reached nodes as participants in the feature. + +Reuse retained parent Evidence when the server advertises it, or supply complete +Evidence inline. Include Hopper or Ghidra Evidence only when its artifact digest +matches exactly. + +## Recover an oversized display An oversized response reports `resource_constraint` and `details.resource: "transport"`. Reuse `details.reported_limits.evidence_reference` with `inspect_analysis_view` for a -summary, one module, or a stable page of module identities. Use -`trace_application_feature` once a module, route, or string seed is known. +summary, one module, or a stable page of module identities. `export_evidence_bundle` writes the complete canonical session to a caller-selected file. These workflows keep the original Evidence and coverage; a broad follow-up can also exceed framing. +A client can truncate a successful result inside its text envelope without a +REA `resource_constraint`. Treat a truncation notice or incomplete JSON as a +display limitation; do not claim the full record was examined. Reuse the known +parent ID for smaller views, or export complete Evidence to a selected file +while the session remains open. If the parent ID was lost from the preview, +use the prior analysis summary or explicit ledger entries to recover it; do +not invent an ID or rerun analysis just to reproduce the same display. + BrowserWindow preferences, preload and contextBridge surfaces, IPC registrations, utility processes, and native binding requests are static syntax observations. Only a unique exact literal IPC channel match supports an inferred @@ -22,13 +92,6 @@ pairing. Dynamic or ambiguous channels remain unresolved. A requested `.node` member is not a verified native export. Never claim runtime reachability, registration, defaults, or policy enforcement from static analysis. -Use `trace_application_feature` on existing application Evidence for one -literal node ID, route, string, API, IPC channel, module, or native export. -Choose a direction and reuse the returned application Evidence ID when the -connected server advertises retained references, or supply complete Evidence -inline. Include -Hopper or Ghidra Evidence only when its artifact digest matches exactly. - For version comparison, analyze each version once, then call `compare_application_versions`. Accept only its digest, source-map, structural fingerprint, or non-module semantic matches. Module ordinals and minified names diff --git a/docs/installation.md b/docs/installation.md index da726182b7..1463d7aec1 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -142,6 +142,18 @@ listed after the table because its connector is not one of these files: | Pi | `pi` | | Hermes | `hermes` | +For GitHub Copilot CLI 1.0.95, `gpt-4.1` model metadata can block ordinary +chat with REA's full input-schema profile before sending a model request +(`compaction_static_context_blocked`; [#1554](https://github.com/morluto/rea/issues/1554)). +The existing compact profile passed native chat and analysis with the complete +tool inventory on Linux using a loopback model fixture. After setup, add +`"REA_MCP_INPUT_SCHEMA_PROFILE": "compact"` to the `env` object of the +`mcpServers.rea` entry in Copilot's `mcp-config.json`, preserving its other +settings, then restart the client. Compact advertisements can reduce nested +schema constraints; REA still validates complete canonical inputs. Full-profile +`gpt-5.4` metadata is also verified locally. Live model API acceptance and +Windows remain unverified; see [native client verification](testing.md). + For [Pi](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/mcp.md), setup writes a `type: "stdio"` entry under `mcpServers.rea` in the user-level `~/.pi/agent/mcp.json` and installs the bundled skill under `~/.agents/skills`. diff --git a/docs/process-capture.md b/docs/process-capture.md index 63810d629f..772b396020 100644 --- a/docs/process-capture.md +++ b/docs/process-capture.md @@ -114,6 +114,12 @@ from launch, must be ordered, and must fall within `timeout_ms`. Inputs marked Environment overrides are recorded in the scenario commitment; inherited values are not copied into Evidence. +Terminal dimensions must satisfy `columns × (rows + scrollback) ≤ 1,000,000`. +Every scheduled resize uses the same selected scrollback and must fit this +working-buffer budget. REA rejects oversized scenarios before launching the +command; it never clamps the requested dimensions. This limit is independent +of `limits.output_bytes`, which bounds retained terminal observations. + The capture contains raw PTY output chunks and rendered terminal states, interaction dispatch outcomes, exit reason, sampled process-tree observations, and settlement status. When filesystem paths are selected, REA records diff --git a/docs/testing.md b/docs/testing.md index ecb2600b6e..c1e53717a4 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -160,10 +160,21 @@ chat or `REA_VERIFY_RUNTIME_ROOT` for a production-only installed package. The native offline BYOK adapter uses a loopback OpenAI completions/SSE fixture and `gpt-5.4` model metadata; model inference and token usage are synthetic. Set `REA_VERIFY_COPILOT_MODEL` to examine another model configuration. In the -verified client, `gpt-4.1` blocks the complete catalog before HTTP with +verified client, `gpt-4.1` blocks the full input-schema profile before HTTP with `compaction_static_context_blocked`, even when requesting a larger BYOK prompt -capacity. The effective capacity is unknown; this lane does not establish a -fix for that client/model limit. It preserves the complete catalog and schemas. +capacity. The effective capacity is unknown. The default verifier preserves +the complete catalog and full schemas. To verify the existing compact profile: + +```bash +REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- chat +REA_VERIFY_COPILOT_MODEL=gpt-4.1 REA_VERIFY_COPILOT_SCHEMA_PROFILE=compact npm run verify:copilot-client -- call +``` + +Both native workflows passed with the complete tool inventory. The verifier selects the +profile only in its disposable Copilot registration and records it in the +receipt. Compact advertisements can reduce nested validation structure; the +server still validates complete canonical inputs. This is a tested alternate +workflow, not a fix for the full-profile client/model limit tracked in #1554. Live model APIs, native Windows and default OS-home skill discovery remain unverified. @@ -1053,6 +1064,39 @@ real local Codex CLI with: npm run verify:agent ``` +The lane runs scenarios sequentially in standalone Codex (`--no-daemon`) with a +disposable HOME and CODEX_HOME under `~/.cache/rea-agent-evaluations/`. Set +`REA_AGENT_EVAL_ROOT` to choose another parent directory outside the OS temporary +directory: current Codex refuses helper aliases under `/tmp`, preventing shell +commands and skill reads even if MCP works. Each owned run directory is removed +after verification unless fixture retention is requested. The lane +installs the packaged skill and MCP registration through public `rea setup`, +rather than copying a skill into a project that also inherits the caller's +skills and configuration. Only authentication is copied, when present, and it +is removed even with `REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration, +rules, plugins or history is copied; shell snapshots and additional agents are +disabled. Existing API-key environment authentication remains available. +The native missing-provider scenario preapproves only `open_binary` and +`close_binary` in the disposable client so it reaches REA's actual provider +boundary instead of stopping at Codex approval. This does not validate a real +native engine or authorize arbitrary runtime execution. + +Select a smaller run with `REA_AGENT_EVAL_SCENARIOS`, a comma-separated list of +scenario IDs. `asar-zh` exercises the packaged desktop rubric from a Chinese +request. `javascript-module-view` exercises summary analysis, module paging and +item inspection in a tree with unrelated modules; its routing and text checks +remain heuristic, so review the retained transcript for selector choice and +coverage claims. Set `REA_CODEX_CLI` to an installed Codex executable and +`REA_AGENT_EVAL_MODEL` to an account-supported model. Use an external resource +limit for the whole process tree when testing on a constrained host; a Node +heap limit alone does not bound client and child-process memory. + +`missing-target` leaves desktop fixtures visible without selecting one in the +request. It requires zero REA calls and a structured request for an app name or +artifact path. Its `expectedFirstTool` is `null`, and +`targetClarificationPassed` grades that limited routing outcome independently +of analysis prose heuristics. Analyzing a nearby example cannot pass this case. + The packaged desktop and parser-comparison scenarios now assess a closed set of known fixture claims. Each final answer must be a strict JSON manifest containing exactly the requested claim IDs, values, Evidence IDs, and Evidence authority and @@ -1077,9 +1121,21 @@ navigation-context, and address-context scenarios have no factual rubric and remain `not_assessed`; the managed workflow requires both artifact and member inspection to answer its type and entry-point question. +The closed factual scenarios request complete producer Evidence on the initial +analysis so their source selectors can authenticate all configured facts. +Summary/page/item workflows are exercised separately by `javascript-module-view`; +its heuristic gate does not establish full-producer factual correctness. + All scenarios retain routing, workflow, validation, repetition, process-exit, and token-use gates. Configured factual scenarios additionally require a factual pass. Their answer-text heuristics are diagnostic and do not affect the gate. +Validation counts include complete text-only REA `invalid_request` envelopes +and failed SDK calls whose arguments violate the named current REA contract. +Client approval rejections and provider unavailability remain distinct failures; +truncated JSON previews are not parsed into invented error codes. +Repeated `binary_session` observations separated by a successful `open_binary` +or `close_binary` are treated as lifecycle verification. An unchanged retry, +including one after a failed lifecycle call, still counts as repetition. Scenarios without a rubric retain the legacy text-heuristic gate. `answerTermCoverageMet` checks case-insensitive substrings, `epistemicCuePresent` checks keywords, and `finalCitesEvidence` checks only an ID's @@ -1087,7 +1143,11 @@ presence. These metrics can still accept fabricated prose and must not be read as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete tool results and final answers for review. -Report schema version 3 changes the top-level `factualCorrectness` from a constant +Report schema version 4 adds negative routing scenarios with nullable +`expectedFirstTool` and a `targetClarificationPassed` outcome. Consumers must +allow an expected absence of REA calls; this does not make an analysis scenario +pass without its required tools. Version 3 changed the top-level +`factualCorrectness` from a constant string to an assessment summary with status, scope, and assessed/passed/failed/ not-assessed scenario counts. Scenario records include the factual assessment and configured claim IDs. The evaluation scope is diff --git a/scripts/verify-agent-experience.mjs b/scripts/verify-agent-experience.mjs index 513835860b..99ab060ddf 100644 --- a/scripts/verify-agent-experience.mjs +++ b/scripts/verify-agent-experience.mjs @@ -1,9 +1,18 @@ import { spawn } from "node:child_process"; -import { cp, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; +import { + chmod, + copyFile, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; +import { homedir } from "node:os"; +import { delimiter, dirname, join, resolve } from "node:path"; import { createInterface } from "node:readline"; import { fileURLToPath } from "node:url"; +import { z } from "zod"; import { evaluateCodexEvents } from "../dist/evaluation/CodexAgentEval.js"; import { @@ -15,32 +24,95 @@ import { agentEvaluationPassed, summarizeFactualCorrectness, } from "../dist/evaluation/AgentEvaluationReport.js"; -import { MCP_STARTUP_POLICY } from "../dist/mcpStartupPolicy.js"; +import { PRODUCT_IDENTITY } from "../dist/identity.js"; import { completeVerifierRun, createVerifierRun } from "./lib/verifier-run.mjs"; const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const verifierRun = createVerifierRun(); -const evaluationRoot = await mkdtemp(join(tmpdir(), "rea-agent-eval-")); -const fixtureRoot = join(evaluationRoot, "targets"); -const skillDestination = join( - evaluationRoot, - ".agents/skills/reverse-engineer-anything", -); const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000); const codex = process.env.REA_CODEX_CLI ?? "codex"; const optionalModel = process.env.REA_AGENT_EVAL_MODEL; if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000) throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000"); -const codexVersion = await readCommandVersion(codex); +// Codex refuses helper aliases when CODEX_HOME is under the OS temp directory. +// Keep the disposable account in a private cache, then remove the owned run. +const evaluationParent = resolve( + process.env.REA_AGENT_EVAL_ROOT ?? + join(homedir(), ".cache", "rea-agent-evaluations"), +); +await mkdir(evaluationParent, { recursive: true, mode: 0o700 }); +const evaluationRoot = await mkdtemp(join(evaluationParent, "run-")); +const fixtureRoot = join(evaluationRoot, "targets"); +const account = join(evaluationRoot, "account"); +const codexHome = join(evaluationRoot, "codex"); +const authPath = join(codexHome, "auth.json"); +const skillDestination = join( + account, + ".agents/skills/reverse-engineer-anything", +); +const evaluationEnvironment = { + ...Object.fromEntries( + Object.entries(process.env).filter( + ([name]) => !/^(?:REA_|CODEX_|GHIDRA_|HOPPER_|IDA_)/u.test(name), + ), + ), + HOME: account, + USERPROFILE: account, + CODEX_HOME: codexHome, + XDG_CONFIG_HOME: join(account, ".config"), + XDG_DATA_HOME: join(account, ".local/share"), + XDG_CACHE_HOME: join(account, ".cache"), + REA_PROCESS_RUN_ID: verifierRun.run_id, + PATH: `${dirname(process.execPath)}${delimiter}${process.env.PATH ?? ""}`, +}; try { - await mkdir(fixtureRoot, { recursive: true }); - await cp( - join(repositoryRoot, "skills/reverse-engineer-anything"), - skillDestination, - { recursive: true }, + const codexVersion = await readCommandVersion(codex); + await Promise.all([ + mkdir(fixtureRoot, { recursive: true }), + mkdir(account), + mkdir(codexHome, { mode: 0o700 }), + ]); + // Reuse authentication only. User config, plugins, rules, history and skills + // must not make a fresh installed-skill trial pass accidentally. + try { + await copyFile( + join(process.env.CODEX_HOME ?? join(homedir(), ".codex"), "auth.json"), + authPath, + ); + await chmod(authPath, 0o600); + } catch (cause) { + if (cause?.code !== "ENOENT") throw cause; + // Environment/API-key authentication can work without a local auth file. + } + await writeFile( + join(codexHome, "config.toml"), + 'approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n[features]\nmulti_agent = false\nshell_snapshot = false\n[history]\npersistence = "none"\n', + { mode: 0o600 }, ); + for (const approval of ["--dry-run", "--yes"]) { + await runProcess( + process.execPath, + [ + join(repositoryRoot, "scripts/rea.mjs"), + "setup", + "--client", + "codex", + approval, + "--json", + ], + evaluationRoot, + 120_000, + evaluationEnvironment, + ); + } + const installedSkill = await readFile( + join(skillDestination, "SKILL.md"), + "utf8", + ); + if (!installedSkill.includes(` version: "${PRODUCT_IDENTITY.skillVersion}"`)) + throw new Error("Setup did not install the current packaged REA skill"); const requestedScenarioIds = new Set( (process.env.REA_AGENT_EVAL_SCENARIOS ?? "") .split(",") @@ -57,7 +129,11 @@ try { expectedFirstTool: "open_binary", requiresEvidence: false, requiredAnswerTermGroups: [["unavailable", "could not", "provider"]], - prompt: `Explain what the shipped native program at ${targets.native} does. Base the answer on artifact analysis, and clearly state any facts that remain unavailable.`, + // The verifier authorizes these named session operations, not arbitrary + // writes or runtime execution. Missing-engine behavior stays observable. + approvedTools: ["open_binary", "close_binary"], + requiredToolSubsequence: ["open_binary", "close_binary"], + prompt: `Explain what static artifact evidence establishes about the shipped native program at ${targets.native}. Do not execute the target or install tools. Clearly state unavailable provider evidence, and close any analysis session you opened before finishing.`, }, { id: "asar", @@ -70,6 +146,38 @@ try { ], prompt: `Explain how the desktop application at ${targets.javascript} exposes APIs to its renderer. Base the answer on the shipped application artifact, and state what remains unknown.`, }, + { + id: "asar-zh", + fixtureScenarioId: "asar", + expectedFirstTool: "analyze_javascript_application", + requiresEvidence: true, + requiredAnswerTermGroups: [["profileapi"], ["profile:read"]], + prompt: `請靜態分析已交付的 Electron 程式 ${targets.javascript}，說明 renderer 如何透過 preload 讀取 profile，以及 main handler 的關係。不要執行目標或安裝分析引擎；引用完整的產出 Evidence，保留 runtime 未知事項。`, + }, + { + id: "javascript-module-view", + expectedFirstTool: "analyze_javascript_application", + requiresEvidence: true, + requiredToolSubsequence: [ + "analyze_javascript_application", + "inspect_analysis_view", + "inspect_analysis_view", + ], + requiredAnswerTermGroups: [ + ["main.js"], + ["node", "identity"], + ["observations", "recorded"], + ], + prompt: `Inspect the recorded module identity, exports and observations for main.js in the shipped JavaScript tree ${targets.largeJavascript}. Use REA without directly reading target files or executing the target. Keep results focused on that module, cite Evidence, and state the limits of what its module view establishes.`, + }, + { + id: "missing-target", + expectedFirstTool: null, + requiresEvidence: false, + requiredAnswerTermGroups: [], + prompt: + 'Reverse engineer my desktop app and explain how it saves user settings. Use static analysis; do not install software. End with one JSON object containing status ("needs_target" or "analyzed"), requested_input ("app_name_or_artifact_path" or null), and question (a string or null).', + }, { id: "javascript-export-shape", expectedFirstTool: "analyze_javascript_application", @@ -145,12 +253,14 @@ try { const results = []; for (const scenario of scenarios) { process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`); - const fixtureClaims = agentFixtureClaims(scenario.id, targets); - const answerInstructions = agentFixtureAnswerInstructions(scenario.id); + const fixtureId = scenario.fixtureScenarioId ?? scenario.id; + const fixtureClaims = agentFixtureClaims(fixtureId, targets); + const answerInstructions = agentFixtureAnswerInstructions(fixtureId); const execution = await runCodex( answerInstructions.length === 0 ? scenario.prompt - : `${scenario.prompt}\n\n${answerInstructions}`, + : `${scenario.prompt}\n\nThis closed factual rubric needs complete producer Evidence in the transcript. Request complete detail on each initial application analysis when supported; summary or view records alone are not accepted by this rubric.\n\n${answerInstructions}`, + scenario.approvedTools ?? [], ); const transcriptDirectory = process.env.REA_AGENT_EVAL_TRANSCRIPT_DIR; if (transcriptDirectory !== undefined) { @@ -175,6 +285,12 @@ try { ...(fixtureClaims === undefined ? {} : { fixtureClaims }), }, ); + // Fixtures are deliberately visible beside prior investigation artifacts. + // Their existence does not identify the unnamed app in this request. + const targetClarificationPassed = + scenario.expectedFirstTool === null + ? validTargetClarification(metrics.finalMessage) + : undefined; const result = { id: scenario.id, expectedFirstTool: scenario.expectedFirstTool, @@ -185,6 +301,9 @@ try { exitCode: execution.exitCode, stderr: execution.stderr, ...metrics, + ...(targetClarificationPassed === undefined + ? {} + : { targetClarificationPassed }), }; results.push(result); process.stderr.write( @@ -194,13 +313,20 @@ try { const factualCorrectness = summarizeFactualCorrectness(results); const summary = { - schemaVersion: 3, + schemaVersion: 4, evaluationScope: "routing_workflow_and_configured_fixture_claims", factualCorrectness, verifier_run: await completeVerifierRun(verifierRun), codex, codexVersion, model: optionalModel ?? null, + installation: { + method: "rea_setup_codex", + isolatedHome: true, + isolatedCodexHome: true, + skillVersion: PRODUCT_IDENTITY.skillVersion, + authentication: "caller_auth_only_removed_after_run", + }, scenarios: results, totals: { scenarios: results.length, @@ -253,12 +379,32 @@ try { `Agent routing/workflow/answer checks failed: ${failed.map(({ id }) => id).join(", ")}`, ); } finally { + // Even a deliberately retained fixture directory must never retain auth. + await rm(authPath, { force: true }); if (process.env.REA_AGENT_EVAL_KEEP_FIXTURES !== "true") await rm(evaluationRoot, { recursive: true, force: true }); } async function createTargets(root, includeManaged) { const fixtures = await createAgentEvaluationFixtures(root, repositoryRoot); + const largeJavascript = join(root, "large-app"); + await mkdir(join(largeJavascript, "lib"), { recursive: true }); + await Promise.all([ + writeFile( + join(largeJavascript, "package.json"), + '{"name":"large-app","main":"main.js"}\n', + ), + writeFile( + join(largeJavascript, "main.js"), + 'const { ipcMain } = require("electron");\nipcMain.handle("profile:read", (_event, id) => ({ id }));\n', + ), + ...Array.from({ length: 180 }, (_, index) => + writeFile( + join(largeJavascript, "lib", `formatter-${String(index)}.js`), + `export const format = (value) => String(value).trim();\n`, + ), + ), + ]); const managedOutput = join(root, "managed-output"); if (includeManaged) { @@ -292,15 +438,29 @@ async function createTargets(root, includeManaged) { return { native: "/bin/true", ...fixtures, + largeJavascript, managed: join(managedOutput, "AgentEval.dll"), }; } -async function runCodex(prompt) { +function validTargetClarification(message) { + const schema = z.strictObject({ + status: z.literal("needs_target"), + requested_input: z.literal("app_name_or_artifact_path"), + question: z.string().refine((value) => value.trim().length > 0), + }); + try { + return schema.safeParse(JSON.parse(message)).success; + } catch (cause) { + if (cause instanceof SyntaxError) return false; + throw cause; + } +} + +async function runCodex(prompt, approvedTools) { const arguments_ = [ + "--no-daemon", "exec", - "--ignore-user-config", - "--ignore-rules", "--ephemeral", "--skip-git-repo-check", "--sandbox", @@ -312,18 +472,16 @@ async function runCodex(prompt) { evaluationRoot, "-c", 'approval_policy="never"', - "-c", - `mcp_servers.rea.command=${JSON.stringify(process.execPath)}`, - "-c", - `mcp_servers.rea.args=${JSON.stringify([join(repositoryRoot, "scripts/rea.mjs"), "mcp"])}`, - "-c", - `mcp_servers.rea.startup_timeout_sec=${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)}`, + ...approvedTools.flatMap((name) => [ + "-c", + `mcp_servers.rea.tools.${name}.approval_mode="approve"`, + ]), ...(optionalModel === undefined ? [] : ["--model", optionalModel]), prompt, ]; const child = spawn(codex, arguments_, { cwd: evaluationRoot, - env: process.env, + env: evaluationEnvironment, stdio: ["ignore", "pipe", "pipe"], }); const events = []; @@ -362,9 +520,16 @@ async function runCodex(prompt) { return { events, exitCode, stderr: stderr.join("").slice(0, 32_768) }; } -async function runProcess(command, arguments_, cwd, timeout) { +async function runProcess( + command, + arguments_, + cwd, + timeout, + env = process.env, +) { const child = spawn(command, arguments_, { cwd, + env, stdio: ["ignore", "ignore", "pipe"], }); let stderr = ""; diff --git a/scripts/verify-copilot-client.mjs b/scripts/verify-copilot-client.mjs index 1160710314..e3980d269e 100644 --- a/scripts/verify-copilot-client.mjs +++ b/scripts/verify-copilot-client.mjs @@ -22,6 +22,12 @@ const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo; const command = process.env.REA_VERIFY_COPILOT_COMMAND ?? "copilot"; const mode = process.argv[2] ?? "call"; const model = process.env.REA_VERIFY_COPILOT_MODEL ?? "gpt-5.4"; +const inputSchemaProfile = + process.env.REA_VERIFY_COPILOT_SCHEMA_PROFILE ?? "full"; +assert( + ["full", "compact"].includes(inputSchemaProfile), + "REA_VERIFY_COPILOT_SCHEMA_PROFILE must be full or compact", +); assert( ["chat", "call"].includes(mode), "Usage: verify-copilot-client.mjs [chat|call]", @@ -311,6 +317,16 @@ try { assert.deepEqual(JSON.parse(repeated.stdout).appliedActions, []); assert.equal(await readFile(configPath, "utf8"), configured); assert.equal(await readFile(`${configPath}.rea.backup`, "utf8"), original); + if (inputSchemaProfile === "compact") { + // Select the existing presentation in this disposable native-client profile. + const customized = JSON.parse(configured); + assert(customized.mcpServers.rea); + customized.mcpServers.rea.env = { + ...customized.mcpServers.rea.env, + REA_MCP_INPUT_SCHEMA_PROFILE: inputSchemaProfile, + }; + await writeFile(configPath, JSON.stringify(customized, null, 2)); + } const installedSkill = await readFile( join(skillDirectory, "SKILL.md"), "utf8", @@ -408,6 +424,7 @@ try { clientVersion, mode, model, + inputSchemaProfile, passed: failure === undefined, artifacts: lab, runtimeRoot, @@ -421,7 +438,9 @@ try { skillDiscovery: "native COPILOT_HOME MCP configuration and caller-added isolated skill directory; default OS-home discovery unverified", capacityBoundary: - "gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified local fixture configuration", + inputSchemaProfile === "compact" + ? "Explicit compact profile: complete inventory and canonical server validation; advertised schemas may be reduced. This does not fix the full-profile gpt-4.1 context boundary." + : "gpt-4.1 model configuration blocks the full catalog before HTTP even with a ten-million-token requested prompt override; gpt-5.4 is the verified full-profile local fixture configuration", catalogSize, schemasChecked, spillReadRequested, diff --git a/src/android/AndroidTargetSnapshot.ts b/src/android/AndroidTargetSnapshot.ts index 1b372a6dcd..f49f65f9c1 100644 --- a/src/android/AndroidTargetSnapshot.ts +++ b/src/android/AndroidTargetSnapshot.ts @@ -20,10 +20,12 @@ export const snapshotAndroidTarget = async ( sha256: string, root: string, operation: AndroidOperation, + signal?: AbortSignal, ): Promise => { const snapshot = join(root, "target.apk"); try { - await copyAndroidSnapshot(path, snapshot, sha256); + await copyAndroidSnapshot(path, snapshot, sha256, signal); + signal?.throwIfAborted(); await chmod(snapshot, 0o400); return snapshot; } catch (cause: unknown) { @@ -44,10 +46,12 @@ export const snapshotAndroidEngine = async ( sha256: string, root: string, operation: AndroidOperation, + signal?: AbortSignal, ): Promise<{ path: string; sha256: string }> => { const path = join(root, "engine.jar"); try { - const observed = await copyAndroidSnapshot(source, path, sha256); + const observed = await copyAndroidSnapshot(source, path, sha256, signal); + signal?.throwIfAborted(); await chmod(path, 0o400); return { path, sha256: observed }; } catch (cause: unknown) { @@ -65,15 +69,22 @@ const copyAndroidSnapshot = async ( sourcePath: string, snapshotPath: string, expectedSha256: string, + signal?: AbortSignal, ): Promise => { - const source = await openRegularFile(sourcePath, { symlinks: "follow" }); + signal?.throwIfAborted(); + const source = await openRegularFile(sourcePath, { + symlinks: "follow", + signal, + }); let destination: FileHandle | undefined; let inputStream: ReadStream | undefined; let outputStream: WriteStream | undefined; + let removeSnapshot = false; let copiedBytes = 0; const hash = createHash("sha256"); try { const initial = await source.stat(); + signal?.throwIfAborted(); const digest = new Transform({ transform( chunk: Buffer, @@ -95,13 +106,15 @@ const copyAndroidSnapshot = async ( }); try { destination = await open(snapshotPath, "wx", 0o600); + signal?.throwIfAborted(); inputStream = source.createReadStream({ start: 0, end: initial.size, autoClose: false, }); outputStream = destination.createWriteStream({ autoClose: false }); - await pipeline(inputStream, digest, outputStream); + await pipeline(inputStream, digest, outputStream, { signal }); + signal?.throwIfAborted(); await verifySourceState(source, sourcePath, initial, copiedBytes); const observedSha256 = hash.digest("hex"); if (observedSha256 !== expectedSha256) @@ -110,8 +123,7 @@ const copyAndroidSnapshot = async ( ); return observedSha256; } catch (cause: unknown) { - if (destination !== undefined) - await rm(snapshotPath, { force: true }).catch(() => undefined); + removeSnapshot = destination !== undefined; throw cause; } } finally { @@ -120,7 +132,12 @@ const copyAndroidSnapshot = async ( // descriptor closure, after the admitted source state has been verified. inputStream?.destroy(); outputStream?.destroy(); - await Promise.all([source.close(), destination?.close()]); + try { + await Promise.all([source.close(), destination?.close()]); + } finally { + if (removeSnapshot) + await rm(snapshotPath, { force: true }).catch(() => undefined); + } } }; diff --git a/src/android/JadxConfiguration.ts b/src/android/JadxConfiguration.ts index 8c9d29357c..b44d1f28bf 100644 --- a/src/android/JadxConfiguration.ts +++ b/src/android/JadxConfiguration.ts @@ -52,7 +52,11 @@ export const inspectJadxAvailability = async ( signal?: AbortSignal, ): Promise => { try { - const configuration = await readJadxConfiguration(environment, signal); + const configuration = await readJadxConfigurationInputs( + environment, + signal, + ); + await validateJadxConfigurationContents(configuration, environment, signal); return { status: "available", code: null, @@ -77,52 +81,83 @@ export const inspectJadxAvailability = async ( } }; -/** Resolve caller-selected tools and preserve the same prerequisite diagnostics. */ -export const resolveJadxConfiguration = async ( +/** Resolve caller-selected paths and JVM options without probing their contents. */ +export const resolveJadxConfigurationInputs = async ( environment: Readonly>, operation: AndroidOperation, signal?: AbortSignal, ): Promise => { try { - return await readJadxConfiguration(environment, signal); + return await readJadxConfigurationInputs(environment, signal); } catch (cause) { if (signal?.aborted === true) throw cause; if (!(cause instanceof JadxConfigurationFailure)) throw cause; - const output = cause.diagnostics; - throw new AnalysisCapabilityUnavailableError( - "jadx", - operation, - cause.message, - { - cause, - userMessage: cause.message, - ...(typeof output.stdout === "string" && - typeof output.stderr === "string" - ? { - capturedOutput: { - stdout: output.stdout, - stderr: output.stderr, - truncated: output.output_truncated === true, - }, - } - : {}), - }, - ); + throw projectConfigurationFailure(operation, cause); } }; -const readJadxConfiguration = async ( +/** Probe Java and archive prerequisites before starting a new engine session. */ +export const validateJadxConfiguration = async ( + configuration: JadxConfiguration, + environment: Readonly>, + operation: AndroidOperation, + signal?: AbortSignal, +): Promise => { + try { + await validateJadxConfigurationContents(configuration, environment, signal); + } catch (cause) { + if (signal?.aborted === true) throw cause; + if (!(cause instanceof JadxConfigurationFailure)) throw cause; + throw projectConfigurationFailure(operation, cause); + } +}; + +const readJadxConfigurationInputs = async ( environment: Readonly>, signal?: AbortSignal, ): Promise => { + signal?.throwIfAborted(); requireSupportedHost(); const jar = await resolveJar(environment); const java = await resolveJava(environment, signal); await requireJavaHomeExecutable(environment.JAVA_HOME, java); - const jvmArguments = resolveJvmArguments(environment); - await inspectJavaRuntime(java, environment, signal); - await requireEngineArchive(jar, signal); - return { jar, java, jvmArguments }; + signal?.throwIfAborted(); + return { jar, java, jvmArguments: resolveJvmArguments(environment) }; +}; + +const validateJadxConfigurationContents = async ( + configuration: JadxConfiguration, + environment: Readonly>, + signal?: AbortSignal, +): Promise => { + signal?.throwIfAborted(); + await inspectJavaRuntime(configuration.java, environment, signal); + await requireEngineArchive(configuration.jar, signal); +}; + +const projectConfigurationFailure = ( + operation: AndroidOperation, + cause: JadxConfigurationFailure, +): AnalysisCapabilityUnavailableError => { + const output = cause.diagnostics; + return new AnalysisCapabilityUnavailableError( + "jadx", + operation, + cause.message, + { + cause, + userMessage: cause.message, + ...(typeof output.stdout === "string" && typeof output.stderr === "string" + ? { + capturedOutput: { + stdout: output.stdout, + stderr: output.stderr, + truncated: output.output_truncated === true, + }, + } + : {}), + }, + ); }; const requireEngineArchive = async ( diff --git a/src/android/JadxProvider.ts b/src/android/JadxProvider.ts index af2e0e7003..6fda5a143e 100644 --- a/src/android/JadxProvider.ts +++ b/src/android/JadxProvider.ts @@ -27,7 +27,8 @@ import { hashStableFile } from "../filesystem/StableFileHash.js"; import { RegularFileChangedError } from "../filesystem/RegularFile.js"; import { inspectJadxAvailability, - resolveJadxConfiguration, + resolveJadxConfigurationInputs, + validateJadxConfiguration, } from "./JadxConfiguration.js"; import type { JadxLauncher } from "./JadxMcpTransport.js"; @@ -333,7 +334,7 @@ export class JadxProvider implements AndroidAnalysisPort { request.operation, `Target ${target.path} is ${target.format}; provide one standalone APK.`, ); - const configuration = await resolveJadxConfiguration( + const configuration = await resolveJadxConfigurationInputs( this.environment, request.operation, signal, @@ -382,18 +383,26 @@ export class JadxProvider implements AndroidAnalysisPort { } let retained = this.#retained; if (retained === undefined) { + await validateJadxConfiguration( + configuration, + this.environment, + request.operation, + signal, + ); root = await PrivateRuntimeRoot.create({ prefix: "rea-android-" }); const engine = await snapshotAndroidEngine( configuration.jar, jarHash, root.path, request.operation, + signal, ); const snapshot = await snapshotAndroidTarget( target.path, target.sha256, root.path, request.operation, + signal, ); const bridge = join(root.path, "ReaJadxBridge.java"); await copyFile(BRIDGE_SOURCE, bridge); diff --git a/src/application/EvidenceBundleCommands.ts b/src/application/EvidenceBundleCommands.ts index 928b794def..5f220e3a1e 100644 --- a/src/application/EvidenceBundleCommands.ts +++ b/src/application/EvidenceBundleCommands.ts @@ -5,27 +5,22 @@ import type { JsonValue } from "../domain/jsonValue.js"; import { jsonValueSchema } from "../domain/jsonValue.js"; import { err, ok, type Result } from "../domain/result.js"; import { compareBundles } from "../domain/bundleComparison.js"; -import { EvidenceLedger } from "./investigation/EvidenceLedger.js"; import { readEvidenceBundle, writeEvidenceBundle, } from "./EvidenceBundleFiles.js"; -/** Validate and merge one bundle using the session Evidence ledger. */ +/** Validate a standalone bundle and report its complete record/revision counts. */ export const importEvidenceBundleCommand = async ( path: string, ): Promise> => { const loaded = await readEvidenceBundle(path); if (!loaded.ok) return loaded; - const ledger = createLedger(); - const imported = ledger.import(loaded.value); - return imported.ok - ? ok({ - imported: imported.value.recordsAdded, - unknowns_added: imported.value.unknownsAdded, - total: ledger.export().records.length, - }) - : imported; + return ok({ + imported: loaded.value.records.length, + unknowns_added: loaded.value.unknowns.length, + total: loaded.value.records.length, + }); }; /** Validate a source bundle and atomically export canonical bytes. */ @@ -64,8 +59,6 @@ export const compareEvidenceBundlesCommand = async (input: { } }; -const createLedger = (): EvidenceLedger => new EvidenceLedger(); - const projectWrite = ( bundle: EvidenceBundle, written: Awaited>, diff --git a/src/application/ReconstructionObligationCandidates.test.ts b/src/application/ReconstructionObligationCandidates.test.ts index ed3ac93cef..2d01d9c7ee 100644 --- a/src/application/ReconstructionObligationCandidates.test.ts +++ b/src/application/ReconstructionObligationCandidates.test.ts @@ -1,3 +1,11 @@ +import { javascriptApplicationAnalysisResultSchema } from "../domain/javascript/javascriptApplicationAnalysis.js"; +import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../contracts/javascript/javascriptRuntimeReconciliationExample.js"; +import { + createJavaScriptSemanticGraph, + createJavaScriptSemanticGraphNode, + JavaScriptSemanticEvidenceContextRegistry, +} from "../domain/javascript/javascriptSemanticGraph.js"; +import { proofEvidence } from "./ReconstructionObligationLedger.fixture.js"; import { describe, expect, it } from "vitest"; import { createEvidence } from "../domain/evidence.js"; @@ -171,3 +179,115 @@ describe("reconstruction obligation candidates", () => { ); }); }); + +it("keeps each semantic candidate bound to its own file and authority context", () => { + const contexts = new JavaScriptSemanticEvidenceContextRegistry(); + const base = javascriptApplicationAnalysisResultSchema.parse( + JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result, + ); + const sourceRecords = [ + proofEvidence("first-file"), + proofEvidence("second-file"), + ]; + const module = createJavaScriptSemanticGraphNode( + { + kind: "module", + identity: { + artifact_sha256: base.root_artifact_sha256, + module_path: "app.js", + source_range: null, + role_key: "module", + }, + function_node_id: null, + application_node_ids: [], + label: "app.js", + properties: {}, + evidence: artifactEvidence(base.root_artifact_sha256, "app.js"), + }, + contexts, + ); + const requests = sourceRecords.flatMap((source, fileIndex) => + [1, 2].map((line) => { + const sha256 = String(fileIndex + 4).repeat(64); + const path = `file-${String(fileIndex)}.js`; + return createJavaScriptSemanticGraphNode( + { + kind: "request", + identity: { + artifact_sha256: sha256, + module_path: path, + source_range: { + start: { line, column: 0 }, + end: { line, column: 8 }, + }, + role_key: `request-${String(line)}`, + }, + function_node_id: null, + application_node_ids: [], + label: `${path}:${String(line)}`, + properties: {}, + evidence: { + ...artifactEvidence(sha256, path, "ast-static-analysis"), + evidence_ids: [source.evidence_id], + }, + }, + contexts, + ); + }), + ); + const { graph_id: _graphId, ...semantic } = base.semantic_graph; + const semanticGraph = createJavaScriptSemanticGraph({ + ...semantic, + root_node_ids: [module.node_id], + nodes: [module, ...requests], + evidence_contexts: contexts.contexts, + }); + const example = JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE; + const application = createEvidence( + { + path: base.input_path, + sha256: base.root_artifact_sha256, + format: base.format, + }, + example.provider, + { + predicateType: example.predicate_type, + operation: example.operation, + parameters: example.parameters, + result: jsonValueSchema.parse({ ...base, semantic_graph: semanticGraph }), + confidence: "derived", + authority: "shipped-artifact", + evidenceLinks: sourceRecords.map(({ evidence_id }) => evidence_id), + }, + ); + const generated = deriveReconstructionObligationCandidates( + createEvidenceBundle([...sourceRecords, application]), + [], + ); + expect(generated.limitations).toEqual([]); + expect(generated.candidates).toHaveLength(requests.length); + for (const [index, node] of requests.entries()) { + const source = sourceRecords[Math.floor(index / 2)]; + expect( + generated.candidates.find( + ({ target }) => target.semantic_node_id === node.node_id, + ), + ).toMatchObject({ + title: node.label, + family: "request", + source_state: "candidate", + target: { + artifact_sha256: node.identity.artifact_sha256, + semantic_node_id: node.node_id, + }, + authority_references: [ + { + evidence_id: source?.evidence_id, + authority: "controlled-replay", + state: "candidate", + location: `${semanticGraph.graph_id}/node/${node.node_id}`, + }, + ], + }); + } +}); diff --git a/src/application/ReconstructionObligationCandidates.ts b/src/application/ReconstructionObligationCandidates.ts index e099527698..60eaaed135 100644 --- a/src/application/ReconstructionObligationCandidates.ts +++ b/src/application/ReconstructionObligationCandidates.ts @@ -12,7 +12,6 @@ import type { ReconstructionObligation, ReviewedReconstructionObligation, } from "../domain/reconstructionObligationLedgerSchemas.js"; -import { resolveJavaScriptSemanticEvidence } from "../domain/javascript/javascriptSemanticGraph.js"; import { parseApplicationGraphEvidence } from "./javascript/JavaScriptApplicationEvidenceGraph.js"; import { applicationObligationPolicy, @@ -115,14 +114,21 @@ const deriveApplicationCandidates = ( }), ); } - if (source.semanticGraph !== null) + if (source.semanticGraph !== null) { + const evidenceContexts = new Map( + source.semanticGraph.evidence_contexts.map((context) => [ + context.context_id, + context, + ]), + ); for (const node of source.semanticGraph.nodes) { const candidatePolicy = semanticObligationPolicy(node.kind); if (candidatePolicy === undefined) continue; - const evidenceContext = resolveJavaScriptSemanticEvidence( - source.semanticGraph, - node.evidence, - ); + const evidenceContext = evidenceContexts.get(node.evidence.context_id); + if (evidenceContext === undefined) + throw new TypeError( + "Semantic evidence reference names an absent context", + ); addCandidate( candidates, generatedCandidate({ @@ -144,6 +150,7 @@ const deriveApplicationCandidates = ( }), ); } + } if (source.graph.coverage.status !== "complete") limitations.add( `Application graph ${source.graph.graph_id} coverage is ${source.graph.coverage.status}; omitted candidates remain unknown.`, diff --git a/src/application/SetupSkill.ts b/src/application/SetupSkill.ts index 25228d14ba..50d0d96f9b 100644 --- a/src/application/SetupSkill.ts +++ b/src/application/SetupSkill.ts @@ -10,6 +10,7 @@ import { clientSkillDirectories } from "./SupportedClients.js"; export const MANAGED_SKILL_FILES = [ "SKILL.md", + "references/connection-and-recovery.md", "references/native-and-artifacts.md", "references/javascript-applications.md", "references/android-applications.md", diff --git a/src/application/analysisView/AnalysisViewService.test.ts b/src/application/analysisView/AnalysisViewService.test.ts index 0ad7917dea..aa74776624 100644 --- a/src/application/analysisView/AnalysisViewService.test.ts +++ b/src/application/analysisView/AnalysisViewService.test.ts @@ -11,6 +11,7 @@ import { } from "../../../tests/fixtures/analysisView.js"; import { inspectAnalysisView, + inspectAnalysisViewValidated, summarizeRetainedAnalysis, } from "./AnalysisViewService.js"; import type { Evidence } from "../../domain/evidence.js"; @@ -58,6 +59,35 @@ it.each(["/fixtures/unknown.exe", ""])( }, ); +it("reports inline Evidence identity failures at the complete source path", () => { + const parent = analysisViewLayoutEvidence(); + const input = { + source: { + kind: "inline", + evidence: { ...parent, evidence_id: `ev_${"0".repeat(64)}` }, + }, + view: { kind: "summary" }, + } as const; + + for (const result of [ + inspectAnalysisView(input), + inspectAnalysisViewValidated(input), + ]) + expect(result).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisInputError", + issues: [ + { + path: ["source", "evidence", "evidence_id"], + reason: "invalid_value", + message: "Evidence semantic identifier does not match its record", + }, + ], + }, + }); +}); + it("projects a small native view from an authenticated >10 MiB retained dossier", () => { const original = functionDossierSchema.parse(ghidraFunctionDossier()); const parent = createEvidence( diff --git a/src/application/analysisView/AnalysisViewService.ts b/src/application/analysisView/AnalysisViewService.ts index ea079802ec..380feb59e9 100644 --- a/src/application/analysisView/AnalysisViewService.ts +++ b/src/application/analysisView/AnalysisViewService.ts @@ -132,8 +132,11 @@ const resolveAnalysisViewSource = ( return err( analysisInputErrorFromIssues( INSPECT_ANALYSIS_VIEW_OPERATION, - cause.issues, - source.evidence, + cause.issues.map((issue) => ({ + ...issue, + path: ["source", "evidence", ...issue.path], + })), + { source }, { cause }, ), ); diff --git a/src/application/javascript/JavaScriptSemanticGraphBuilder.test.ts b/src/application/javascript/JavaScriptSemanticGraphBuilder.test.ts index 216aa309cd..590f4950b4 100644 --- a/src/application/javascript/JavaScriptSemanticGraphBuilder.test.ts +++ b/src/application/javascript/JavaScriptSemanticGraphBuilder.test.ts @@ -355,7 +355,7 @@ const semanticNodeFor = (roleKey: string) => { unpacked: false, text: { included: true, value: "" }, }; - const state = createSemanticGraphProjectionState({ nodes: [] }); + const state = createSemanticGraphProjectionState(); return constructSemanticGraphNode( file, { @@ -565,7 +565,7 @@ it("does not mark a file truncated when it exactly fills its budget", () => { it("only records node loss when the budget rejects a new identity", () => { // The truncation signal must record blocked node creation, not a zero // remaining budget: a file using exactly its share drops nothing. - const state = createSemanticGraphProjectionState({ nodes: [] }); + const state = createSemanticGraphProjectionState(); state.fileNodeBudget = 2; state.fileNodesDropped = false; const first = addSemanticGraphNode(state, semanticNodeFor("first")); diff --git a/src/application/javascript/JavaScriptSemanticGraphBuilder.ts b/src/application/javascript/JavaScriptSemanticGraphBuilder.ts index a0cc8ccf8e..3ed0cafdae 100644 --- a/src/application/javascript/JavaScriptSemanticGraphBuilder.ts +++ b/src/application/javascript/JavaScriptSemanticGraphBuilder.ts @@ -148,7 +148,7 @@ export interface JavaScriptSemanticGraphProjection { /** Accumulate semantic nodes while each source IR is still file-local. */ export const createJavaScriptSemanticGraphProjection = (): JavaScriptSemanticGraphProjection => { - const state = emptyState({ nodes: [] }); + const state = emptyState(); const fingerprints: JavaScriptSemanticFingerprint[] = []; let truncatedFiles = 0; const projectSourceSteps = function* ( diff --git a/src/application/javascript/JavaScriptSemanticGraphConstruction.ts b/src/application/javascript/JavaScriptSemanticGraphConstruction.ts index ebe27fe43d..c61047d5bf 100644 --- a/src/application/javascript/JavaScriptSemanticGraphConstruction.ts +++ b/src/application/javascript/JavaScriptSemanticGraphConstruction.ts @@ -28,7 +28,6 @@ export interface SemanticGraphProjectionState { readonly unknowns: Map; readonly evidenceContexts: JavaScriptSemanticEvidenceContextRegistry; readonly roots: Set; - readonly applicationNodeIdsByLocation: ReadonlyMap; /** * Remaining node budget for the file currently being projected. * @@ -69,32 +68,29 @@ export interface SemanticRelationConstructionInput { readonly properties?: Readonly>; } -/** Create empty projection state with structural application-node mappings. */ -export const createSemanticGraphProjectionState = ( - applicationGraph: Pick, -): SemanticGraphProjectionState => { - const index = indexApplicationNodes(applicationGraph.nodes); - return { +/** Create empty projection state; application-node links are bound at finish. */ +export const createSemanticGraphProjectionState = + (): SemanticGraphProjectionState => ({ nodes: new Map(), relations: new Map(), unknowns: new Map(), evidenceContexts: new JavaScriptSemanticEvidenceContextRegistry(), roots: new Set(), - applicationNodeIdsByLocation: index.identifiers, - }; -}; + }); /** Bind retained semantic identities after all application observations exist. */ export const bindSemanticGraphApplicationNodes = ( state: SemanticGraphProjectionState, applicationGraph: Pick, ): void => { - const index = indexApplicationNodes(applicationGraph.nodes).identifiers; + const applicationNodeIdsByLocation = indexApplicationNodes( + applicationGraph.nodes, + ); for (const [identifier, node] of state.nodes) { state.nodes.set(identifier, { ...node, application_node_ids: [ - ...(index.get( + ...(applicationNodeIdsByLocation.get( applicationLocationKey( node.identity.artifact_sha256, node.identity.module_path, @@ -122,7 +118,7 @@ export const constructSemanticGraphNode = ( role_key: input.roleKey, }, function_node_id: input.functionNodeId, - application_node_ids: matchingApplicationNodeIds(file, input, state), + application_node_ids: [], label: input.label === null ? null : javascriptDisplayText(input.label), properties: input.properties ?? {}, evidence: observedSemanticEvidence(file, input.location), @@ -247,9 +243,7 @@ export const addSemanticFallbackRoot = ( const indexApplicationNodes = ( nodes: JavaScriptApplicationGraph["nodes"], -): { - readonly identifiers: ReadonlyMap; -} => { +): ReadonlyMap => { const identifiers = new Map>(); for (const node of nodes) { for (const observation of node.observations) { @@ -280,19 +274,9 @@ const indexApplicationNodes = ( return [key, sorted]; }), ); - return { identifiers: indexed }; + return indexed; }; -const matchingApplicationNodeIds = ( - file: JavaScriptArtifactFile, - input: SemanticNodeConstructionInput, - state: SemanticGraphProjectionState, -): string[] => [ - ...(state.applicationNodeIdsByLocation.get( - applicationLocationKey(file.sha256, file.path, input.location), - ) ?? []), -]; - const addApplicationNodeIdentifier = ( identifiers: Map>, key: string, diff --git a/src/artifacts/ArtifactPaths.test.ts b/src/artifacts/ArtifactPaths.test.ts index 4834cf2d4e..2300c63bbb 100644 --- a/src/artifacts/ArtifactPaths.test.ts +++ b/src/artifacts/ArtifactPaths.test.ts @@ -60,6 +60,30 @@ describe("English Unicode case collisions", () => { ]); }); + it("retains every spelling in a high-cardinality folded sibling group", () => { + const count = 8_192; + const variants = caseVariants("abcdefghijklm", count); + const { limitation, byPath } = annotate(variants); + const ordered = [...variants].sort(); + const first = ordered[0]; + const second = ordered[1]; + const original = variants[0]; + if (first === undefined || second === undefined || original === undefined) { + throw new Error("Expected at least two case variants"); + } + expect(limitation).toBe(ENGLISH_UNICODE_CASE_INVENTORY_LIMITATION); + expect(byPath.size).toBe(count); + expect(byPath.get(first)).toEqual([ + `Logical path ${first} collides under English (en-US) Unicode case folding with ${second} and ${String(count - 2)} other spellings; ${NOTE}`, + ]); + expect(byPath.get(original)?.[0]).toContain( + `with ${first} and ${String(count - 2)} other spellings;`, + ); + expect( + [...byPath.values()].every((limitations) => limitations.length === 1), + ).toBe(true); + }); + it("stays linear for large case-variant subtrees", () => { const count = 5_000; const paths = Array.from({ length: count }, (_, index) => [ @@ -81,3 +105,12 @@ describe("English Unicode case collisions", () => { expect(byPath.get(deep)).toEqual([]); }); }); + +const caseVariants = (lowercase: string, count: number): string[] => + Array.from({ length: count }, (_, mask) => + [...lowercase] + .map((character, index) => + (mask & (1 << index)) === 0 ? character : character.toUpperCase(), + ) + .join(""), + ); diff --git a/src/artifacts/ArtifactPaths.ts b/src/artifacts/ArtifactPaths.ts index 3950b4c6e2..ea81ea41fa 100644 --- a/src/artifacts/ArtifactPaths.ts +++ b/src/artifacts/ArtifactPaths.ts @@ -177,12 +177,12 @@ const englishUnicodeCaseCollisions = ( if (path.length === 0 || path === ".") continue; let node = root; for (const part of path.split("/")) { - const folded = englishCaseFold(part); - const spellings = node.spellingsByFold.get(folded); - if (spellings === undefined) node.spellingsByFold.set(folded, [part]); - else if (!spellings.includes(part)) spellings.push(part); let child = node.bySpelling.get(part); if (child === undefined) { + const folded = englishCaseFold(part); + const spellings = node.spellingsByFold.get(folded); + if (spellings === undefined) node.spellingsByFold.set(folded, [part]); + else spellings.push(part); child = emptyCaseNode(); node.bySpelling.set(part, child); } diff --git a/src/artifacts/AsarArtifactReader.ts b/src/artifacts/AsarArtifactReader.ts index 80e9fec0f2..1408bce8a1 100644 --- a/src/artifacts/AsarArtifactReader.ts +++ b/src/artifacts/AsarArtifactReader.ts @@ -19,7 +19,7 @@ import { isAbsolute, join, relative, sep } from "node:path"; import { Readable, Transform } from "node:stream"; import { pipeline } from "node:stream/promises"; -import { getRawHeader, listPackage, statFile, uncache } from "@electron/asar"; +import { listPackage, statFile, uncache } from "@electron/asar"; import { ArtifactReaderFailure, @@ -41,6 +41,11 @@ export class AsarArtifactReader implements ArtifactReader { #archiveSize: number | undefined; #headerSize: number | undefined; readonly #entries = new Map(); + readonly #unpackedHandles = new Map(); + readonly #unpackedHandleClosePromises = new Map< + UnpackedFileHandle, + Promise + >(); #snapshot: { readonly path: string; readonly sha256: string } | undefined; #snapshotRoot: string | undefined; @@ -65,6 +70,7 @@ export class AsarArtifactReader implements ArtifactReader { try { const admitted = await admitAsarHeader(headerHandle); this.#archiveSize = admitted.archiveBytes; + this.#headerSize = admitted.headerBytes; } finally { await headerHandle.close(); } @@ -72,7 +78,6 @@ export class AsarArtifactReader implements ArtifactReader { paths = listPackage(snapshotPath, { isPack: false }).sort((left, right) => left.localeCompare(right, "en"), ); - this.#headerSize = getRawHeader(snapshotPath).headerSize; const archiveMetadata = await lstat(snapshotPath); if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink()) throw new ArtifactReaderFailure( @@ -156,32 +161,8 @@ export class AsarArtifactReader implements ArtifactReader { ); const { metadata } = state; const producedEntry = state.entry; - if (producedEntry.unpacked) { - const handle = await this.#openUnpackedEntry(producedEntry); - try { - const observed = await handle.stat(); - abortIfNeeded(signal); - if (!observed.isFile()) - throw new ArtifactReaderFailure( - "path", - `ASAR unpacked entry is not a regular file: ${producedEntry.path}`, - ); - } catch (cause: unknown) { - await handle.close().catch(() => undefined); - throw asarFailure(this.path, `read ${producedEntry.path}`, cause); - } - const source = handle.createReadStream({ - start: 0, - autoClose: true, - }); - return readValidatedAsarEntry( - source, - undefined, - producedEntry.path, - this.path, - signal, - ); - } + if (producedEntry.unpacked) + return this.#openUnpackedStream(producedEntry, signal); const archiveSize = this.#archiveSize; const headerSize = this.#headerSize; const offset = @@ -315,22 +296,39 @@ export class AsarArtifactReader implements ArtifactReader { } async close(): Promise { + let cleanupFailure: ArtifactReaderFailure | undefined; + for (const [handle, path] of this.#unpackedHandles) { + try { + await this.#closeUnpackedHandle(handle); + } catch (cause: unknown) { + cleanupFailure = ArtifactReaderFailure.withCleanup( + cleanupFailure ?? cause, + ArtifactReaderFailure.cleanupObservation(cause, path), + ); + } + } const root = this.#snapshotRoot; if (root !== undefined) { try { await this.removeSnapshot(root); + if (this.#snapshotRoot === root) this.#snapshotRoot = undefined; } catch (cause: unknown) { - throw ArtifactReaderFailure.withCleanup(cause, { - reason: `Could not remove ASAR snapshot: ${root}`, - resources: [root], - }); + cleanupFailure = ArtifactReaderFailure.withCleanup( + cleanupFailure ?? cause, + { + reason: `Could not remove ASAR snapshot: ${root}`, + resources: [root], + }, + ); } - this.#snapshotRoot = undefined; } - if (this.#snapshot !== undefined) uncache(this.#snapshot.path); - this.#snapshot = undefined; - this.#resetArchiveState(); - this.#entries.clear(); + if (root === undefined || this.#snapshotRoot === undefined) { + if (this.#snapshot !== undefined) uncache(this.#snapshot.path); + this.#snapshot = undefined; + this.#resetArchiveState(); + this.#entries.clear(); + } + if (cleanupFailure !== undefined) throw cleanupFailure; } #snapshotPath(): string { @@ -346,8 +344,12 @@ export class AsarArtifactReader implements ArtifactReader { return []; } - async #openUnpackedEntry(entry: ArtifactEntry): Promise { + async #openUnpackedStream( + entry: ArtifactEntry, + signal?: AbortSignal, + ): Promise { const unpackedRoot = `${this.path}.unpacked`; + let handle: UnpackedFileHandle | undefined; try { const rootMetadata = await lstat(unpackedRoot); if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink()) @@ -374,22 +376,18 @@ export class AsarArtifactReader implements ArtifactReader { "path", `ASAR unpacked entry is not a regular file: ${entry.path}`, ); - const handle = await this.openUnpackedFile( + handle = await this.openUnpackedFile( canonical, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK, ); - const openedMetadata = await handle - .stat() - .catch(async (cause: unknown) => { - await handle.close().catch(() => undefined); - throw cause; - }); + const openedHandle = handle; + this.#unpackedHandles.set(openedHandle, canonical); + const openedMetadata = await openedHandle.stat(); if ( !openedMetadata.isFile() || openedMetadata.dev !== pathMetadata.dev || openedMetadata.ino !== pathMetadata.ino ) { - await handle.close(); throw new ArtifactReaderFailure( "integrity", `ASAR unpacked entry changed before read: ${entry.path}`, @@ -402,25 +400,54 @@ export class AsarArtifactReader implements ArtifactReader { }, ); } - return handle; + abortIfNeeded(signal); + const source = openedHandle.createReadStream({ + start: 0, + autoClose: false, + }); + source.once("close", () => { + void this.#closeUnpackedHandle(openedHandle).catch(() => undefined); + }); + return readValidatedAsarEntry( + source, + undefined, + entry.path, + this.path, + signal, + ); } catch (cause: unknown) { - if (cause instanceof ArtifactReaderFailure) throw cause; - if (entry.unpacked && isMissingFile(cause)) - throw new ArtifactReaderFailure( - "unavailable", - `ASAR unpacked entry bytes are unavailable: ${entry.path}`, - { cause }, - { - logicalPath: entry.path, - declaredSha256: entry.declaredSha256, - calculatedSha256: null, - unpacked: true, - }, - ); - throw asarFailure(this.path, `read ${entry.path}`, cause); + let failure = asarUnpackedEntryFailure(this.path, entry, cause); + if (handle !== undefined && this.#unpackedHandles.has(handle)) { + try { + await this.#closeUnpackedHandle(handle); + } catch (cleanupCause: unknown) { + failure = ArtifactReaderFailure.withCleanup( + failure, + ArtifactReaderFailure.cleanupObservation( + cleanupCause, + this.#unpackedHandles.get(handle) ?? entry.path, + ), + ); + } + } + throw failure; } } + async #closeUnpackedHandle(handle: UnpackedFileHandle): Promise { + if (!this.#unpackedHandles.has(handle)) return; + const pending = this.#unpackedHandleClosePromises.get(handle); + if (pending !== undefined) return pending; + const closing = Promise.resolve() + .then(() => handle.close()) + .then(() => { + this.#unpackedHandles.delete(handle); + }) + .finally(() => this.#unpackedHandleClosePromises.delete(handle)); + this.#unpackedHandleClosePromises.set(handle, closing); + return closing; + } + #resetArchiveState(): void { this.#archiveSize = undefined; this.#headerSize = undefined; @@ -442,6 +469,27 @@ type OpenAsarUnpackedFile = ( flags: number, ) => Promise; +const asarUnpackedEntryFailure = ( + archivePath: string, + entry: ArtifactEntry, + cause: unknown, +): ArtifactReaderFailure => { + if (cause instanceof ArtifactReaderFailure) return cause; + if (entry.unpacked && isMissingFile(cause)) + return new ArtifactReaderFailure( + "unavailable", + `ASAR unpacked entry bytes are unavailable: ${entry.path}`, + { cause }, + { + logicalPath: entry.path, + declaredSha256: entry.declaredSha256, + calculatedSha256: null, + unpacked: true, + }, + ); + return asarFailure(archivePath, `read ${entry.path}`, cause); +}; + const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({ ...entry, limitations: [...entry.limitations], diff --git a/src/artifacts/SafeOutputTree.ts b/src/artifacts/SafeOutputTree.ts index d9610192eb..8d32ac5dcb 100644 --- a/src/artifacts/SafeOutputTree.ts +++ b/src/artifacts/SafeOutputTree.ts @@ -1,12 +1,11 @@ import { createHash } from "node:crypto"; import { constants } from "node:fs"; import { - lstat, mkdir, open, readdir, realpath, - rm, + rmdir, type FileHandle, } from "node:fs/promises"; import { basename, dirname, isAbsolute, join, resolve } from "node:path"; @@ -19,6 +18,19 @@ import { normalizeArtifactPath, } from "./ArtifactPaths.js"; import { ArtifactReaderFailure } from "./ArtifactReader.js"; +import { removeOwnedTree } from "./SafeOutputTreeCleanup.js"; +import { + assertHandleIdentity, + assertFilePathIdentity, + assertPathIdentity, + isAbsent, + pathHasIdentity, + readDirectoryIdentity, + readFileIdentity, + replacedDirectory, + type DirectoryIdentity, + type FileIdentity, +} from "./SafeOutputTreeIdentity.js"; /** One file durably written to an operation-owned output tree. */ export interface SafeOutputFile { @@ -36,10 +48,18 @@ export type SafeOutputCleanup = readonly residualPaths: readonly [string, ...string[]]; }; -/** Symlink-resistant materialization in an exclusively owned, initially absent tree. */ +/** + * Materialize files in a tree created by this operation. + * + * Path identities are revalidated around operations, but Node has no portable + * descriptor-relative traversal; a syscall-boundary pathname race remains. + */ export class SafeOutputTree { readonly #registry = new ArtifactPathRegistry(); readonly #outputRoot: string; + readonly #rootIdentity: DirectoryIdentity; + readonly #nestedDirectories = new Map(); + readonly #ownedFiles = new Map(); #published = false; #cleanup: SafeOutputCleanup = { status: "not-required", @@ -47,9 +67,11 @@ export class SafeOutputTree { private constructor( outputRoot: string, + rootIdentity: DirectoryIdentity, private readonly platform: NodeJS.Platform, ) { this.#outputRoot = outputRoot; + this.#rootIdentity = rootIdentity; } /** @@ -91,29 +113,45 @@ export class SafeOutputTree { ); throw cause; }); + let rootIdentity: DirectoryIdentity | undefined; try { + rootIdentity = await readDirectoryIdentity(canonicalOutput); if (platform !== "win32") { const stagingHandle = await open( canonicalOutput, constants.O_RDONLY | constants.O_DIRECTORY, ); try { + await assertHandleIdentity( + stagingHandle, + rootIdentity, + canonicalOutput, + ); await stagingHandle.chmod(0o700); } finally { await stagingHandle.close(); } } - return new SafeOutputTree(canonicalOutput, platform); + await assertPathIdentity(canonicalOutput, rootIdentity); + return new SafeOutputTree(canonicalOutput, rootIdentity, platform); } catch (cause: unknown) { let removalFailure: unknown; + let ownedRootAtPath = false; try { - await rm(canonicalOutput, { recursive: true, force: true }); + ownedRootAtPath = + rootIdentity !== undefined && + (await pathHasIdentity(canonicalOutput, rootIdentity)); + if (ownedRootAtPath) await rmdir(canonicalOutput); + else + removalFailure = new Error( + "Extraction output root identity changed during setup", + ); } catch (cleanupCause: unknown) { removalFailure = cleanupCause; } let absent = false; try { - absent = await isAbsent(canonicalOutput); + absent = ownedRootAtPath && (await isAbsent(canonicalOutput)); } catch (cleanupCause: unknown) { removalFailure ??= cleanupCause; } @@ -122,7 +160,7 @@ export class SafeOutputTree { reason: removalFailure instanceof Error ? removalFailure.message - : "Extraction output root remains after setup failure", + : "Extraction output root ownership could not be verified after setup failure", resources: [canonicalOutput], }); throw cause; @@ -153,7 +191,11 @@ export class SafeOutputTree { ); const path = normalizeArtifactPath(relativePath); this.#registry.add(path, "file"); - const destination = await this.#prepareParent(path); + const lineage = await this.#prepareParent(path); + const parent = lineage.at(-1); + if (parent === undefined) throw replacedDirectory(this.#outputRoot); + const fileName = path.slice(path.lastIndexOf("/") + 1); + const destination = join(parent.path, fileName); const handle = await open( destination, constants.O_CREAT | @@ -162,20 +204,24 @@ export class SafeOutputTree { constants.O_NOFOLLOW, 0o600, ).catch(async (cause: unknown) => { - await throwIfDestinationCaseCollision( - dirname(destination), - path, - cause, - ); + await throwIfDestinationCaseCollision(parent.path, path, cause); throw new ArtifactReaderFailure( "path", `Could not exclusively create extraction path: ${path}`, { cause }, ); }); + const fileIdentity = await readFileIdentity(handle, destination).catch( + async (cause: unknown) => { + await handle.close().catch(() => undefined); + throw cause; + }, + ); + this.#ownedFiles.set(destination, fileIdentity); const hash = createHash("sha256"); let bytes = 0; try { + await this.#assertLineage(lineage); for await (const raw of source) { abortIfNeeded(signal); const chunk = streamChunkToBuffer(raw); @@ -199,9 +245,13 @@ export class SafeOutputTree { "integrity", `Extracted content disagrees with inventory: ${path}`, ); + await this.#assertLineage(lineage); + await assertFilePathIdentity(destination, fileIdentity); await handle.sync(); await handle.close(); const readback = await hashFile(destination, bytes, signal); + await this.#assertLineage(lineage); + await assertFilePathIdentity(destination, fileIdentity); if (readback.sha256 !== sha256 || readback.bytes !== bytes) throw new ArtifactReaderFailure( "integrity", @@ -226,6 +276,7 @@ export class SafeOutputTree { /** Sync the owned output tree and prevent further writes through this instance. */ async commit(): Promise { this.#assertWritable(); + await this.#assertOwnedDirectories(); // Windows has no directory fsync; file contents are already synced in write(). if (this.platform === "win32") { this.#published = true; @@ -241,6 +292,7 @@ export class SafeOutputTree { this.#outputRoot, constants.O_RDONLY | constants.O_DIRECTORY, ); + await assertHandleIdentity(output, this.#rootIdentity, this.#outputRoot); await output.sync(); await parent.sync(); this.#published = true; @@ -257,14 +309,26 @@ export class SafeOutputTree { /** Remove only this operation's unsealed tree and verify absence. */ async rollback(): Promise { - if (this.#published) return structuredClone(this.#cleanup); + if (this.#published || this.#cleanup.status === "complete") + return structuredClone(this.#cleanup); + this.#cleanup = { + status: "incomplete", + residualPaths: [basename(this.#outputRoot)], + }; let removalFailure: unknown; try { - await rm(this.#outputRoot, { recursive: true, force: true }); + await this.#assertOwnedDirectories(); + await removeOwnedTree({ + outputRoot: this.#outputRoot, + rootIdentity: this.#rootIdentity, + directories: this.#nestedDirectories, + files: this.#ownedFiles, + }); } catch (cause: unknown) { removalFailure = cause; } - const absent = await isAbsent(this.#outputRoot); + const absent = + removalFailure === undefined && (await isAbsent(this.#outputRoot)); this.#cleanup = absent ? { status: "complete", residualPaths: [] } : { @@ -289,36 +353,61 @@ export class SafeOutputTree { return structuredClone(this.#cleanup); } - async #prepareParent(relativePath: string): Promise { + async #prepareParent( + relativePath: string, + ): Promise { const parts = relativePath.split("/"); - const fileName = parts.pop(); - if (fileName === undefined) + if (parts.pop() === undefined) throw new ArtifactReaderFailure("path", "Invalid extraction path"); let current = this.#outputRoot; let logicalParent = ""; + const lineage: DirectoryLineageEntry[] = [ + { path: this.#outputRoot, identity: this.#rootIdentity }, + ]; + if (parts.length === 0) + await assertPathIdentity(this.#outputRoot, this.#rootIdentity); for (const part of parts) { - const parentDirectory = current; + const parent = lineage.at(-1); + if (parent === undefined) throw replacedDirectory(this.#outputRoot); const logicalPath = logicalParent.length === 0 ? part : `${logicalParent}/${part}`; current = join(current, part); + await assertPathIdentity(parent.path, parent.identity); let created = true; await mkdir(current, { mode: 0o700 }).catch((cause: unknown) => { if (!isAlreadyExists(cause)) throw cause; created = false; }); - if (!created) - await throwIfDestinationCaseCollision(parentDirectory, logicalPath); - const metadata = await lstat(current); - if (!metadata.isDirectory() || metadata.isSymbolicLink()) { - await throwIfDestinationCaseCollision(parentDirectory, logicalPath); - throw new ArtifactReaderFailure( - "path", - `Unsafe extraction parent: ${relativePath}`, - ); + if (!created) { + await throwIfDestinationCaseCollision(parent.path, logicalPath); + const identity = this.#nestedDirectories.get(current); + if (identity === undefined) throw replacedDirectory(current); + await assertPathIdentity(current, identity); + } else { + const identity = await readDirectoryIdentity(current); + this.#nestedDirectories.set(current, identity); } + await assertPathIdentity(parent.path, parent.identity); + const identity = this.#nestedDirectories.get(current); + if (identity === undefined) throw replacedDirectory(current); + await assertPathIdentity(current, identity); + lineage.push({ path: current, identity }); logicalParent = logicalPath; } - return join(current, fileName); + return lineage; + } + + async #assertLineage( + lineage: readonly DirectoryLineageEntry[], + ): Promise { + for (const { path, identity } of lineage) + await assertPathIdentity(path, identity); + } + + async #assertOwnedDirectories(): Promise { + await assertPathIdentity(this.#outputRoot, this.#rootIdentity); + for (const [path, identity] of this.#nestedDirectories) + await assertPathIdentity(path, identity); } #assertWritable(): void { @@ -327,9 +416,19 @@ export class SafeOutputTree { "integrity", "Extraction tree is already committed", ); + if (this.#cleanup.status !== "not-required") + throw new ArtifactReaderFailure( + "integrity", + "Extraction tree cleanup has already started", + ); } } +type DirectoryLineageEntry = { + readonly path: string; + readonly identity: DirectoryIdentity; +}; + const hashFile = async ( path: string, maximum: number, @@ -376,18 +475,6 @@ const writeAll = async ( } }; -const isAbsent = async (path: string): Promise => - lstat(path).then( - () => false, - (cause: unknown) => { - if (isNotFound(cause)) return true; - throw cause; - }, - ); - -const isNotFound = (cause: unknown): boolean => - cause instanceof Error && "code" in cause && cause.code === "ENOENT"; - const isAlreadyExists = (cause: unknown): boolean => cause instanceof Error && "code" in cause && cause.code === "EEXIST"; diff --git a/src/artifacts/SafeOutputTreeCleanup.ts b/src/artifacts/SafeOutputTreeCleanup.ts new file mode 100644 index 0000000000..91e6565d19 --- /dev/null +++ b/src/artifacts/SafeOutputTreeCleanup.ts @@ -0,0 +1,76 @@ +import { readdir, rmdir, unlink } from "node:fs/promises"; +import { dirname, sep } from "node:path"; + +import { ArtifactReaderFailure } from "./ArtifactReader.js"; +import { + assertFilePathIdentity, + assertPathIdentity, + replacedDirectory, + type DirectoryIdentity, + type FileIdentity, +} from "./SafeOutputTreeIdentity.js"; + +/** Remove only tracked files and empty directories, preserving successful progress. */ +export const removeOwnedTree = async (input: { + readonly outputRoot: string; + readonly rootIdentity: DirectoryIdentity; + readonly directories: Map; + readonly files: Map; +}): Promise => { + const { outputRoot, rootIdentity, directories, files } = input; + const filesByDirectory = new Map(); + for (const entry of files) { + const directory = dirname(entry[0]); + const group = filesByDirectory.get(directory) ?? []; + group.push(entry); + filesByDirectory.set(directory, group); + } + const postorder = [...directories].sort( + ([left], [right]) => + right.slice(outputRoot.length).split(sep).length - + left.slice(outputRoot.length).split(sep).length, + ); + for (const [path, identity] of postorder) { + await removeFiles(path, identity, filesByDirectory.get(path) ?? [], files); + const parentPath = dirname(path); + const parentIdentity = + parentPath === outputRoot ? rootIdentity : directories.get(parentPath); + if (parentIdentity === undefined) throw replacedDirectory(parentPath); + await assertPathIdentity(parentPath, parentIdentity); + await assertPathIdentity(path, identity); + await requireEmpty(path); + await rmdir(path); + directories.delete(path); + } + await removeFiles( + outputRoot, + rootIdentity, + filesByDirectory.get(outputRoot) ?? [], + files, + ); + await assertPathIdentity(outputRoot, rootIdentity); + await requireEmpty(outputRoot); + await rmdir(outputRoot); +}; + +const removeFiles = async ( + directory: string, + identity: DirectoryIdentity, + filesInDirectory: readonly (readonly [string, FileIdentity])[], + retainedFiles: Map, +): Promise => { + for (const [path, fileIdentity] of filesInDirectory) { + await assertPathIdentity(directory, identity); + await assertFilePathIdentity(path, fileIdentity); + await unlink(path); + retainedFiles.delete(path); + } +}; + +const requireEmpty = async (directory: string): Promise => { + if ((await readdir(directory)).length > 0) + throw new ArtifactReaderFailure( + "path", + `Unowned entries remain in extraction directory: ${directory}`, + ); +}; diff --git a/src/artifacts/SafeOutputTreeIdentity.ts b/src/artifacts/SafeOutputTreeIdentity.ts new file mode 100644 index 0000000000..d93cda4236 --- /dev/null +++ b/src/artifacts/SafeOutputTreeIdentity.ts @@ -0,0 +1,104 @@ +import type { BigIntStats } from "node:fs"; +import { lstat } from "node:fs/promises"; +import type { FileHandle } from "node:fs/promises"; + +import { ArtifactReaderFailure } from "./ArtifactReader.js"; + +export type DirectoryIdentity = Pick; +export type FileIdentity = DirectoryIdentity; + +export const readDirectoryIdentity = async ( + path: string, +): Promise => { + const metadata = await lstat(path, { bigint: true }); + if (!metadata.isDirectory()) throw replacedDirectory(path); + return { dev: metadata.dev, ino: metadata.ino }; +}; + +export const assertPathIdentity = async ( + path: string, + expected: DirectoryIdentity, +): Promise => { + const metadata = await lstat(path, { bigint: true }); + if ( + !metadata.isDirectory() || + metadata.dev !== expected.dev || + metadata.ino !== expected.ino + ) + throw replacedDirectory(path); +}; + +export const assertHandleIdentity = async ( + handle: FileHandle, + expected: DirectoryIdentity, + path: string, +): Promise => { + const metadata = await handle.stat({ bigint: true }); + if ( + !metadata.isDirectory() || + metadata.dev !== expected.dev || + metadata.ino !== expected.ino + ) + throw replacedDirectory(path); +}; + +export const readFileIdentity = async ( + handle: FileHandle, + path: string, +): Promise => { + const metadata = await handle.stat({ bigint: true }); + if (!metadata.isFile()) + throw new ArtifactReaderFailure( + "path", + `Extraction file identity changed: ${path}`, + ); + return { dev: metadata.dev, ino: metadata.ino }; +}; + +export const assertFilePathIdentity = async ( + path: string, + expected: FileIdentity, +): Promise => { + const metadata = await lstat(path, { bigint: true }); + if ( + !metadata.isFile() || + metadata.dev !== expected.dev || + metadata.ino !== expected.ino + ) + throw new ArtifactReaderFailure( + "path", + `Extraction file identity changed: ${path}`, + ); +}; + +export const pathHasIdentity = async ( + path: string, + expected: DirectoryIdentity, +): Promise => { + try { + await assertPathIdentity(path, expected); + return true; + } catch (cause: unknown) { + if (isNotFound(cause) || cause instanceof ArtifactReaderFailure) + return false; + throw cause; + } +}; + +export const isAbsent = async (path: string): Promise => + lstat(path).then( + () => false, + (cause: unknown) => { + if (isNotFound(cause)) return true; + throw cause; + }, + ); + +export const replacedDirectory = (path: string): ArtifactReaderFailure => + new ArtifactReaderFailure( + "path", + `Extraction directory identity changed: ${path}`, + ); + +const isNotFound = (cause: unknown): boolean => + cause instanceof Error && "code" in cause && cause.code === "ENOENT"; diff --git a/src/artifacts/extraction/ArtifactExtraction.ts b/src/artifacts/extraction/ArtifactExtraction.ts index 5fcfba7205..03f4750700 100644 --- a/src/artifacts/extraction/ArtifactExtraction.ts +++ b/src/artifacts/extraction/ArtifactExtraction.ts @@ -90,20 +90,14 @@ export const materializeArtifactInventory = async ( const selectedIds = new Set( activeOccurrences.map(({ occurrence_id: id }) => id), ); - const occurrences = new Map(); - const neededNodes = new Set(); - collectOccurrences( - snapshot.occurrences, - selectedIds, - occurrences, - neededNodes, + const neededNodes = new Set( + activeOccurrences.map(({ artifact_id: id }) => id), ); const nodes = new Map(); - collectNodes(snapshot.nodes, neededNodes, nodes); - const inventory: LoadedInventory = { + for (const node of snapshot.nodes) + if (neededNodes.has(node.artifact_id)) nodes.set(node.artifact_id, node); + const inventory: ExtractionInventory = { manifest: snapshot.manifest, - occurrences, - nodes, integrityContradictions: snapshot.integrity_contradictions.filter( ({ occurrence_id: id }) => selectedIds.has(id), ), @@ -144,7 +138,7 @@ export const materializeArtifactInventory = async ( "format", `Selected occurrence is not an extractable regular child file: ${occurrence.logical_path} (${occurrence.occurrence_id})`, ); - const node = inventory.nodes.get(occurrence.artifact_id); + const node = nodes.get(occurrence.artifact_id); if (node === undefined) throw new ArtifactReaderFailure( "integrity", @@ -183,7 +177,7 @@ const materializeSelection = async ({ }: { readonly input: ArtifactExtractionInput; readonly sourcePath: string; - readonly inventory: LoadedInventory; + readonly inventory: ExtractionInventory; readonly selected: readonly SelectedOccurrence[]; readonly signal: AbortSignal | undefined; }): Promise => { @@ -298,7 +292,7 @@ const materializeSelection = async ({ const createExtractionResult = ( input: ArtifactExtractionInput, - inventory: LoadedInventory, + inventory: ExtractionInventory, selected: readonly SelectedOccurrence[], extracted: readonly ExtractedOccurrence[], ): ArtifactExtractionResult => { @@ -333,35 +327,11 @@ const createExtractionResult = ( }); }; -interface LoadedInventory { +interface ExtractionInventory { readonly manifest: ArtifactGraphManifest; - readonly occurrences: ReadonlyMap; - readonly nodes: ReadonlyMap; readonly integrityContradictions: readonly IntegrityContradiction[]; } -const collectOccurrences = ( - items: readonly ArtifactOccurrence[], - selected: ReadonlySet, - output: Map, - neededNodes: Set, -): void => { - for (const item of items) { - if (!selected.has(item.occurrence_id)) continue; - output.set(item.occurrence_id, item); - if (item.artifact_id !== null) neededNodes.add(item.artifact_id); - } -}; - -const collectNodes = ( - items: readonly ArtifactNode[], - selected: ReadonlySet, - output: Map, -): void => { - for (const item of items) - if (selected.has(item.artifact_id)) output.set(item.artifact_id, item); -}; - const ZIP_FORMATS = ["ipa", "apk", "msix", "appx", "zip"] as const; const isZipFormat = ( diff --git a/src/artifacts/inventory/ArtifactGraphConstruction.ts b/src/artifacts/inventory/ArtifactGraphConstruction.ts index 4868b9fab1..99d76e9f14 100644 --- a/src/artifacts/inventory/ArtifactGraphConstruction.ts +++ b/src/artifacts/inventory/ArtifactGraphConstruction.ts @@ -295,23 +295,53 @@ export const createArtifactEdges = ( return edges; }; -export const nearestParent = ( - path: string, - occurrences: ReadonlyMap, +interface ArtifactParentIndex { + readonly children: Map; + occurrence?: MutableOccurrence; +} + +/** Resolve strict observed ancestors without rebuilding every path prefix. */ +export const resolveOccurrenceParents = ( + occurrences: readonly MutableOccurrence[], + byPath: ReadonlyMap, expandedContainerIds: ReadonlySet, -): MutableOccurrence | undefined => { - const parts = path.split("/"); - while (parts.length > 1) { - parts.pop(); - const candidate = occurrences.get(parts.join("/")); +): void => { + const root: ArtifactParentIndex = { children: new Map() }; + for (const occurrence of byPath.values()) { if ( - candidate !== undefined && - (candidate.entry_kind === "directory" || - expandedContainerIds.has(candidate.occurrence_id)) + occurrence.entry_kind !== "directory" && + !expandedContainerIds.has(occurrence.occurrence_id) ) - return candidate; + continue; + let branch = root; + for (const segment of occurrence.logical_path.split("/")) { + let child = branch.children.get(segment); + if (child === undefined) { + child = { children: new Map() }; + branch.children.set(segment, child); + } + branch = child; + } + branch.occurrence = occurrence; + } + for (const occurrence of occurrences) { + const path = occurrence.logical_path; + let branch = root; + let parent: MutableOccurrence | undefined; + // Walk parent segments only. Stop scanning the path once no observed + // directory or expanded container can be an ancestor. + for (let start = 0; ;) { + const slash = path.indexOf("/", start); + if (slash < 0) break; + const segment = path.slice(start, slash); + const child = branch.children.get(segment); + if (child === undefined) break; + branch = child; + parent = branch.occurrence ?? parent; + start = slash + 1; + } + occurrence.parent_occurrence_id = parent?.occurrence_id ?? null; } - return undefined; }; /** diff --git a/src/artifacts/inventory/scanReader.ts b/src/artifacts/inventory/scanReader.ts index a68f4030e0..aa2a0b391c 100644 --- a/src/artifacts/inventory/scanReader.ts +++ b/src/artifacts/inventory/scanReader.ts @@ -18,7 +18,7 @@ import { createArtifactNode, createOccurrence, noteUnreadContainerSuffix, - nearestParent, + resolveOccurrenceParents, type MutableOccurrence, } from "./ArtifactGraphConstruction.js"; import { @@ -91,13 +91,11 @@ export const scanReader = async ( await visitArtifactEntries(context, reader, ""); // Archive directories may appear after their children. Resolve containment // against the complete index before directory identities are materialized. - for (const occurrence of occurrences) - occurrence.parent_occurrence_id = - nearestParent( - occurrence.logical_path, - context.occurrenceByPath, - context.expandedContainerIds, - )?.occurrence_id ?? null; + resolveOccurrenceParents( + occurrences, + context.occurrenceByPath, + context.expandedContainerIds, + ); return { nodes, occurrences, diff --git a/src/browser/CdpCaptureEventBodyShapes.ts b/src/browser/CdpCaptureEventBodyShapes.ts index 39f4ab5a5d..53882a8192 100644 --- a/src/browser/CdpCaptureEventBodyShapes.ts +++ b/src/browser/CdpCaptureEventBodyShapes.ts @@ -27,8 +27,7 @@ export const ingestResponseBodyShape = ( invalidResponseBodyShape(state, requestId); return; } - const inferred = inferBodyShape(decoded); - updateResponseBodyShape(state, requestId, inferred.shape); + updateResponseBodyShape(state, requestId, inferJsonShape(decoded)); }; export const requestBodyShape = ( @@ -40,10 +39,10 @@ export const requestBodyShape = ( const body = cdpStringValue(request.postData); if (!isJsonContentType(recordValue(request.headers)) || body === undefined) return { status: "unavailable", request: null, response: null }; - const inferred = inferBodyShape(body); + const shape = inferJsonShape(body); return { - status: inferred.shape === null ? "unavailable" : "included", - request: inferred.shape, + status: shape === null ? "unavailable" : "included", + request: shape, response: null, }; }; @@ -68,10 +67,6 @@ export const updateResponseBodyShape = ( }); }; -const inferBodyShape = (text: string): { readonly shape: JsonShape | null } => { - return { shape: inferJsonShape(text) }; -}; - const invalidResponseBodyShape = ( state: CdpCaptureEventsState, requestId: string, diff --git a/src/browser/CdpTargetSession.ts b/src/browser/CdpTargetSession.ts index 4b140e2eda..0c2b5b24f2 100644 --- a/src/browser/CdpTargetSession.ts +++ b/src/browser/CdpTargetSession.ts @@ -51,18 +51,22 @@ export const closeCdpTargetSession = async ( ): Promise => { const { connection, sessionId } = targetSession; const cleanup = new AbortController(); + const domainSignal = + signal === undefined + ? cleanup.signal + : AbortSignal.any([cleanup.signal, signal]); const timeout = setTimeout(() => cleanup.abort(), 1_000); timeout.unref(); try { if (signal?.aborted !== true) { for (const domain of enabledDomains) { - if (cleanup.signal.aborted) break; + if (domainSignal.aborted) break; try { await connection.send( `${domain}.disable`, {}, sessionId, - cleanup.signal, + domainSignal, ); } catch { // Best-effort domain cleanup; transport close is the definitive boundary. diff --git a/src/browser/CdpWebMcpDiscovery.ts b/src/browser/CdpWebMcpDiscovery.ts index 1cdbd3334f..dd4b3fb6cf 100644 --- a/src/browser/CdpWebMcpDiscovery.ts +++ b/src/browser/CdpWebMcpDiscovery.ts @@ -365,20 +365,30 @@ const ingestWebMcpEvent = (options: WebMcpIngestOptions): void => { transientFrames, resolvedTransientFrames, } = options; + if ( + event.method !== "WebMCP.toolsAdded" && + event.method !== "WebMCP.toolsRemoved" + ) + return; const params = recordValue(event.params); - if (params === undefined) return; + if (params === undefined) { + completeness.exclude("webmcp_tools", "invalid_protocol_value"); + return; + } if (event.method === "WebMCP.toolsRemoved") { - for (const removed of recordsValue(params.tools)) { + for (const removed of webMcpEventTools(params.tools, completeness)) { const name = cdpStringValue(removed.name); const frameId = cdpStringValue(removed.frameId); - if (name === undefined || frameId === undefined) continue; + if (name === undefined || frameId === undefined) { + completeness.exclude("webmcp_tools", "invalid_protocol_value"); + continue; + } for (const [key, tool] of tools) if (tool.name === name && tool.frame_id === frameId) tools.delete(key); } return; } - if (event.method !== "WebMCP.toolsAdded") return; - for (const declared of recordsValue(params.tools)) { + for (const declared of webMcpEventTools(params.tools, completeness)) { const frameId = cdpStringValue(declared.frameId); if (frameId !== undefined && transientFrames.has(frameId)) { if (!resolvedTransientFrames.has(frameId)) continue; @@ -391,6 +401,26 @@ const ingestWebMcpEvent = (options: WebMcpIngestOptions): void => { } }; +const webMcpEventTools = ( + value: unknown, + completeness: CdpCaptureCompleteness, +): UnknownRecord[] => { + if (!Array.isArray(value)) { + completeness.exclude("webmcp_tools", "invalid_protocol_value"); + return []; + } + const records: UnknownRecord[] = []; + for (const item of value) { + const record = recordValue(item); + if (record === undefined) { + completeness.exclude("webmcp_tools", "invalid_protocol_value"); + continue; + } + records.push(record); + } + return records; +}; + const normalizeTool = ( value: UnknownRecord, input: DiscoverWebMcpToolsInput, @@ -403,12 +433,11 @@ const normalizeTool = ( const frameId = cdpStringValue(value.frameId); const name = cdpStringValue(value.name); const frame = frameId === undefined ? undefined : frames.get(frameId); - if ( - frameId === undefined || - name === undefined || - frame?.origin === null || - frame === undefined - ) { + if (frameId === undefined || name === undefined) { + completeness.exclude("webmcp_tools", "invalid_protocol_value"); + return undefined; + } + if (frame?.origin === null || frame === undefined) { completeness.exclude("webmcp_tools", "out_of_target_scope"); return undefined; } diff --git a/src/browser/execution/CdpExecutionCapture.ts b/src/browser/execution/CdpExecutionCapture.ts index ebb675c565..023c3f8e2d 100644 --- a/src/browser/execution/CdpExecutionCapture.ts +++ b/src/browser/execution/CdpExecutionCapture.ts @@ -94,9 +94,9 @@ export const captureWebExecution = async ( sources.belongsToDocument(id), ); const sample = - raw === undefined - ? undefined - : normalizePreciseCoverage(raw, sources, retained); + parsed?.success === true + ? normalizePreciseCoverage(parsed.data, sources, retained) + : undefined; await session.options.progress?.report({ phase: "browser_execution", completed: 2, diff --git a/src/browser/execution/CdpPreciseCoverage.ts b/src/browser/execution/CdpPreciseCoverage.ts index 977ee50f87..95e2cdc0d2 100644 --- a/src/browser/execution/CdpPreciseCoverage.ts +++ b/src/browser/execution/CdpPreciseCoverage.ts @@ -1,34 +1,30 @@ +import type { z } from "zod"; import { AnalysisOutputError } from "../../domain/analysisErrorCore.js"; import type { WebExecution } from "../../domain/webExecution.js"; import type { WebRuntimeSource } from "../../domain/webRuntime.js"; -import { preciseCoverageSchema } from "./CdpRuntimeProtocol.js"; +import type { preciseCoverageSchema } from "./CdpRuntimeProtocol.js"; import type { CdpRuntimeSources } from "./CdpRuntimeSources.js"; /** Validate source-range bounds against exact retained UTF-16 text without flattening nested counts. */ export const normalizePreciseCoverage = ( - raw: unknown, + parsed: z.infer, sources: CdpRuntimeSources, retained: readonly WebRuntimeSource[], ): { readonly coverage: WebExecution["coverage"]; readonly timestamp: number; } => { - const parsed = preciseCoverageSchema.safeParse(raw); - if (!parsed.success) - throw new AnalysisOutputError( - sources.session.operation, - `Malformed precise coverage: ${parsed.error.message}`, - ); + const retainedSources = new Map( + retained.map(({ script_id, source }) => [script_id, source]), + ); const scripts: WebExecution["coverage"]["scripts"] = []; let excluded = 0; - for (const script of parsed.data.result) { + for (const script of parsed.result) { if (!sources.belongsToDocument(script.scriptId)) { excluded += 1; continue; } - const source = retained.find( - (item) => item.script_id === script.scriptId, - )?.source; + const source = retainedSources.get(script.scriptId); scripts.push({ script_id: script.scriptId, reported_url: sources.sourceUrl(script.scriptId, script.url), @@ -57,7 +53,7 @@ export const normalizePreciseCoverage = ( }); } return { - timestamp: parsed.data.timestamp, + timestamp: parsed.timestamp, coverage: { state: "captured", reason: null, diff --git a/src/browser/history/CaptureFailures.test.ts b/src/browser/history/CaptureFailures.test.ts index a3df176485..f7cd77b2cd 100644 --- a/src/browser/history/CaptureFailures.test.ts +++ b/src/browser/history/CaptureFailures.test.ts @@ -20,8 +20,12 @@ it.each([ const stderr = "FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory\r\n"; const cause = new OwnedCommandFailure("process", "Decoder failed", { - stdout: { text: "", bytes: 0 }, - stderr: { text: stderr, bytes: Buffer.byteLength(stderr) }, + stdout: { text: "", bytes: 0, observedBytes: 0 }, + stderr: { + text: stderr, + bytes: Buffer.byteLength(stderr), + observedBytes: Buffer.byteLength(stderr), + }, exitCode, signal, }); @@ -70,8 +74,12 @@ it.each([ format: "har", }); const cause = new OwnedCommandFailure("process", "Decoder failed", { - stdout: { text: "", bytes: 0 }, - stderr: { text: stderr, bytes: Buffer.byteLength(stderr) }, + stdout: { text: "", bytes: 0, observedBytes: 0 }, + stderr: { + text: stderr, + bytes: Buffer.byteLength(stderr), + observedBytes: Buffer.byteLength(stderr), + }, exitCode, signal, }); @@ -128,8 +136,16 @@ it("retains actual command status and both diagnostic streams when cleanup also "process", "Observed command failure", { - stdout: { text: "original provider output", bytes: 24 }, - stderr: { text: "private-marker provider reason", bytes: 30 }, + stdout: { + text: "original provider output", + bytes: 24, + observedBytes: 24, + }, + stderr: { + text: "private-marker provider reason", + bytes: 30, + observedBytes: 30, + }, exitCode: 2, signal: null, }, diff --git a/src/contracts/sessionToolContracts.ts b/src/contracts/sessionToolContracts.ts index 6c610b984b..65c29785fa 100644 --- a/src/contracts/sessionToolContracts.ts +++ b/src/contracts/sessionToolContracts.ts @@ -94,7 +94,7 @@ export const SESSION_TOOL_CONTRACTS = [ ), session( "capture_process_scenario", - "Run one caller-selected command under a PTY and return process capture Evidence with residual unknowns. A command name resolves through the inherited PATH; the working directory defaults to the caller's current directory; host environment variables are inherited with scenario overrides. Environment keys cannot contain '=' or NUL, and REA_PROCESS_RUN_ID is reserved for process ownership. Filesystem snapshots are opt-in through filesystem_observation_paths. The target runs with the current user's permissions; this is not a security sandbox.", + "Run one caller-selected command under a PTY and return process capture Evidence with residual unknowns. The renderer admits at most 1,000,000 combined terminal cells, computed as columns × (rows + scrollback); every resize uses the selected scrollback and must fit the same budget. Supply a smaller terminal shape when that product is too large; dimensions are never clamped. A command name resolves through the inherited PATH; the working directory defaults to the caller's current directory; host environment variables are inherited with scenario overrides. Environment keys cannot contain '=' or NUL, and REA_PROCESS_RUN_ID is reserved for process ownership. Filesystem snapshots are opt-in through filesystem_observation_paths. The target runs with the current user's permissions; this is not a security sandbox.", processScenarioSchema, ), session( diff --git a/src/domain/apple/dyldPaths.ts b/src/domain/apple/dyldPaths.ts index 765bb15d14..e853b9a8a9 100644 --- a/src/domain/apple/dyldPaths.ts +++ b/src/domain/apple/dyldPaths.ts @@ -25,6 +25,7 @@ export const resolveTreePath = async ( let hops = 0; while (pending.length > 0) { const segment = pending.shift() ?? ""; + if (segment === ".") continue; if (segment === "..") { if (resolved.length === 0) return { kind: "escapes" }; resolved.pop(); @@ -50,8 +51,12 @@ export const resolveTreePath = async ( : { kind: "not-file" }; }; -const segmentsOf = (path: string): string[] => - path.split("/").filter((segment) => segment !== "" && segment !== "."); +const segmentsOf = (path: string): string[] => { + const segments = path.split("/").filter((segment) => segment !== ""); + // a trailing slash requires a directory, just like a final dot component. + if (path.endsWith("/")) segments.push("."); + return segments; +}; export const directoryOf = (path: string): string => { const index = path.lastIndexOf("/"); diff --git a/src/domain/apple/dylibResolution.test.ts b/src/domain/apple/dylibResolution.test.ts index 1b262970ad..2d63ce393c 100644 --- a/src/domain/apple/dylibResolution.test.ts +++ b/src/domain/apple/dylibResolution.test.ts @@ -250,6 +250,49 @@ describe("dyld path expansion", () => { }); }); +it("keeps directory requirements in symlink targets", async () => { + const names = [ + "@loader_path/alias-slash", + "@loader_path/alias-dot", + "@loader_path/alias-directory/lib.dylib", + ]; + const trace = await traceDylibLoading( + memoryView( + { + [MAIN]: executable({ + dependencies: names.map((name) => dependency(name)), + }), + "Contents/MacOS/lib.dylib": parsed(slice()), + }, + { + "Contents/MacOS/alias-slash": "lib.dylib/", + "Contents/MacOS/alias-dot": "lib.dylib/.", + "Contents/MacOS/alias-directory": "./", + }, + ), + { roots: [MAIN] }, + ); + expect( + trace.edges.map(({ install_name, candidates, resolution }) => ({ + install_name, + outcome: candidates[0]?.outcome, + resolution, + })), + ).toEqual( + names.map((name, index) => ({ + install_name: name, + outcome: index < 2 ? "absent" : "resolved", + resolution: + index < 2 + ? { status: "unresolved", image: null } + : { status: "resolved", image: "Contents/MacOS/lib.dylib" }, + })), + ); + expect( + trace.findings.map(({ kind, edge_index }) => [kind, edge_index]), + ).toEqual([0, 1].map((index) => ["required-load-unresolved", index])); +}); + describe("dyld resolution outcomes: deriving path and weak-load outcomes", () => { it("keeps paths outside the root undetermined and in-root fallbacks conditional", async () => { const trace = await traceDylibLoading( diff --git a/src/domain/apple/dylibResolution.ts b/src/domain/apple/dylibResolution.ts index 86dc4902e7..871cb7601a 100644 --- a/src/domain/apple/dylibResolution.ts +++ b/src/domain/apple/dylibResolution.ts @@ -4,7 +4,6 @@ import { compatibleSlice, directoryOf, expandPrefix, - joinPath, resolveTreePath, type Expansion, } from "./dyldPaths.js"; @@ -275,7 +274,7 @@ const candidateTemplates = ( path: base.scope === "outside" ? `${base.path.replace(/\/+$/u, "")}/${rest}` - : joinPath(base.path, rest), + : `${base.path}${base.path === "" ? "" : "/"}${rest}`, }; return { expansion, source: "rpath" as const, rpath, rpathOwner: owner }; }), diff --git a/src/domain/firmware/firmwareAnalysis.ts b/src/domain/firmware/firmwareAnalysis.ts index a7cdb35506..0ebbdc2e90 100644 --- a/src/domain/firmware/firmwareAnalysis.ts +++ b/src/domain/firmware/firmwareAnalysis.ts @@ -99,6 +99,8 @@ const chunk = z.strictObject({ export interface FirmwarePartialObservation { readonly kind: "firmware"; readonly result: JsonValue; + /** Completed producer data, artifact identity and analysis metadata. */ + readonly provenance?: JsonValue; } /** Portable results retain reported boundaries, unknowns and extraction lineage. */ diff --git a/src/domain/hopperErrors.ts b/src/domain/hopperErrors.ts index 8cb8490811..f2b37a8a5a 100644 --- a/src/domain/hopperErrors.ts +++ b/src/domain/hopperErrors.ts @@ -16,8 +16,17 @@ export abstract class HopperError extends AnalysisError {} export type HopperLauncherOutcome = { readonly exit_code: number | null; readonly signal: string | null; - readonly stdout: { readonly text: string; readonly bytes: number }; - readonly stderr: { readonly text: string; readonly bytes: number }; + /** Total bytes drained from the stream; retained_bytes reports captured bytes. */ + readonly stdout: { + readonly text: string; + readonly bytes: number; + readonly retained_bytes: number; + }; + readonly stderr: { + readonly text: string; + readonly bytes: number; + readonly retained_bytes: number; + }; readonly output_closed: boolean; readonly diagnostic_truncated: boolean; }; diff --git a/src/domain/javascript/javascriptRuntimeLoadState.ts b/src/domain/javascript/javascriptRuntimeLoadState.ts index 58cf9aeef0..838f606f64 100644 --- a/src/domain/javascript/javascriptRuntimeLoadState.ts +++ b/src/domain/javascript/javascriptRuntimeLoadState.ts @@ -62,9 +62,7 @@ const classifyLayer = ( options.reconciliationComplete && layer.graph.coverage.status === "complete" && scopedEntities.length > 0 && - scopedEntities.every(({ capture }) => - captureScriptsComplete(capture.evidence.evidence_id, runtimeEntities), - ); + scopedEntities.every(({ capture }) => capture.scriptsCompleteWithinScope); const states = nodes.map((node) => { const runtimeNodeIds = uniqueSorted(direct.get(node.node_id) ?? []); if (runtimeNodeIds.length > 0) @@ -134,13 +132,6 @@ const append = ( else current.push(value); }; -const captureScriptsComplete = ( - evidenceId: string, - entities: readonly RuntimeReconciliationEntity[], -): boolean => - entities.find(({ capture }) => capture.evidence.evidence_id === evidenceId) - ?.capture.scriptsCompleteWithinScope === true; - const isLoadStateNode = ( node: ApplicationNode, ): node is ApplicationNode & { diff --git a/src/domain/javascript/javascriptRuntimeReconciliationGraph.ts b/src/domain/javascript/javascriptRuntimeReconciliationGraph.ts index d3c8f371fd..864b9d6f79 100644 --- a/src/domain/javascript/javascriptRuntimeReconciliationGraph.ts +++ b/src/domain/javascript/javascriptRuntimeReconciliationGraph.ts @@ -212,8 +212,11 @@ const graphCoverage = ( const mergeNodes = (nodes: readonly ApplicationNode[]): ApplicationNode[] => { const grouped = new Map(); - for (const node of nodes) - grouped.set(node.node_id, [...(grouped.get(node.node_id) ?? []), node]); + for (const node of nodes) { + const group = grouped.get(node.node_id); + if (group === undefined) grouped.set(node.node_id, [node]); + else group.push(node); + } const merged = [...grouped.values()].map((group) => { const first = group[0]; if (first === undefined) throw new TypeError("Empty JAG node group"); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.scope_depth.test.ts b/src/domain/javascript/javascriptSemanticAnalysis.scope_depth.test.ts new file mode 100644 index 0000000000..175b22cc24 --- /dev/null +++ b/src/domain/javascript/javascriptSemanticAnalysis.scope_depth.test.ts @@ -0,0 +1,43 @@ +import { expect, it, vi } from "vitest"; + +import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js"; +import { onlyBinding } from "./javascriptSemanticAnalysis.fixture.js"; + +it("bounds ancestry reads for repeated deep lexical references", () => { + const ancestryReads = (depth: number): number => { + const original = WeakMap.prototype.get; + let reads = 0; + const get = vi.spyOn(WeakMap.prototype, "get").mockImplementation(function ( + this: WeakMap, + key, + ) { + const value: unknown = original.call(this, key); + if ( + typeof value === "object" && + value !== null && + "type" in value && + typeof value.type === "string" + ) + reads++; + return value; + }); + try { + const ir = analyzeJavaScriptSemantics( + `const a=1; module.exports.value=${"a+(".repeat(depth)}a${")".repeat(depth)};`, + ); + expect(ir.coverage).toEqual({ status: "complete", omittedCount: 0 }); + expect(onlyBinding(ir, "a").value).toEqual({ + status: "literal", + value: 1, + }); + return reads; + } finally { + get.mockRestore(); + } + }; + + const smaller = ancestryReads(80); + const larger = ancestryReads(160); + expect(smaller).toBeGreaterThan(0); + expect(larger).toBeLessThan(3 * smaller); +}); diff --git a/src/domain/javascript/javascriptSemanticAnalysis.ts b/src/domain/javascript/javascriptSemanticAnalysis.ts index 2719c5589e..f0b299b1cb 100644 --- a/src/domain/javascript/javascriptSemanticAnalysis.ts +++ b/src/domain/javascript/javascriptSemanticAnalysis.ts @@ -444,6 +444,7 @@ function* collectDefinitionsSteps( }[] = []; yield* traverseJavaScriptAstSteps(program, { enter: (node, parent, readAncestors) => { + if (parent !== null) state.parentsByNode.set(node, parent); let parentScope = currentSemanticScope(stack); if ( parent !== null && @@ -493,7 +494,10 @@ function* collectDefinitionsSteps( }); } const scope = currentSemanticScope(stack); - state.scopeByNode.set(node, scope); + // Other nodes inherit the nearest boundary through their AST parents. + // Avoid a second full-tree index for the same lexical relationship. + if (parent === null || openedScopes.has(node)) + state.scopeByNode.set(node, scope); collectSemanticCallable({ node, parent, diff --git a/src/domain/javascript/javascriptSemanticCalls.ts b/src/domain/javascript/javascriptSemanticCalls.ts index 2d5f94abd7..53ada16541 100644 --- a/src/domain/javascript/javascriptSemanticCalls.ts +++ b/src/domain/javascript/javascriptSemanticCalls.ts @@ -428,4 +428,10 @@ const enclosingFunction = ( const semanticCallSiteId = ( node: t.CallExpression | t.OptionalCallExpression | t.NewExpression, ): string => - `call:${t.isNewExpression(node) ? "construct" : "call"}:${String(node.start ?? -1)}:${String(node.end ?? -1)}`; + // Join once so retained IDs do not keep a chain of intermediate strings. + [ + "call", + t.isNewExpression(node) ? "construct" : "call", + String(node.start ?? -1), + String(node.end ?? -1), + ].join(":"); diff --git a/src/domain/javascript/javascriptSemanticGraph.ts b/src/domain/javascript/javascriptSemanticGraph.ts index 6908af2a19..ce7eb6ae85 100644 --- a/src/domain/javascript/javascriptSemanticGraph.ts +++ b/src/domain/javascript/javascriptSemanticGraph.ts @@ -19,15 +19,11 @@ import { JAVASCRIPT_SEMANTIC_RELATION_FAMILIES, JAVASCRIPT_SEMANTIC_RELATION_FAMILY, javaScriptSemanticFingerprintInputSchema, - javaScriptSemanticFingerprintSchema, javaScriptSemanticGraphInputSchema, javaScriptSemanticGraphRecordSchema, javaScriptSemanticNodeInputSchema, - javaScriptSemanticNodeSchema, javaScriptSemanticRelationInputSchema, - javaScriptSemanticRelationSchema, javaScriptSemanticUnknownInputSchema, - javaScriptSemanticUnknownSchema, type JavaScriptSemanticGraphInput, type JavaScriptSemanticGraphNode, type JavaScriptSemanticGraphRelation, @@ -132,10 +128,10 @@ export const createJavaScriptSemanticGraphNode = ( application_node_ids: uniqueSorted(parsed.application_node_ids), evidence: evidenceContexts.intern(parsed.evidence), }; - return javaScriptSemanticNodeSchema.parse({ + return { ...semantic, node_id: javaScriptSemanticNodeId(semantic), - }); + }; }; /** Normalize one semantic relationship and derive its exact identifier. */ @@ -148,10 +144,10 @@ export const createJavaScriptSemanticGraphRelation = ( ...parsed, evidence: evidenceContexts.intern(parsed.evidence), }; - return javaScriptSemanticRelationSchema.parse({ + return { ...semantic, relation_id: `jsrg_relation_${digestCanonicalValue(semantic, "JavaScript semantic graph")}`, - }); + }; }; /** Normalize one unresolved semantic frontier and derive its identifier. */ @@ -166,10 +162,10 @@ export const createJavaScriptSemanticGraphUnknown = ( candidate_node_ids: uniqueSorted(parsed.candidate_node_ids), evidence: evidenceContexts.intern(parsed.evidence), }; - return javaScriptSemanticUnknownSchema.parse({ + return { ...semantic, unknown_id: `jsrg_unknown_${digestCanonicalValue(semantic, "JavaScript semantic graph")}`, - }); + }; }; /** Normalize one function fingerprint and derive its component commitment. */ @@ -191,7 +187,7 @@ export const createJavaScriptSemanticFingerprint = ( semantic.components, "JavaScript semantic graph", ); - return javaScriptSemanticFingerprintSchema.parse({ + return { ...semantic, fingerprint_sha256: fingerprintSha256, fingerprint_id: `jsrg_fingerprint_${digestCanonicalValue( @@ -202,7 +198,7 @@ export const createJavaScriptSemanticFingerprint = ( }, "JavaScript semantic graph", )}`, - }); + }; }; type GraphRecord = z.infer; diff --git a/src/domain/javascript/javascriptSemanticGraphQuery.test.ts b/src/domain/javascript/javascriptSemanticGraphQuery.test.ts index 363cf29ad2..17d03a1d1c 100644 --- a/src/domain/javascript/javascriptSemanticGraphQuery.test.ts +++ b/src/domain/javascript/javascriptSemanticGraphQuery.test.ts @@ -4,6 +4,7 @@ import { z } from "zod"; import { digestCanonicalValue } from "../canonicalDigest.js"; import type { ApplicationGraphEvidence } from "./javascriptApplicationEvidenceSchemas.js"; import { + JAVASCRIPT_SEMANTIC_RELATION_FAMILY, JAVASCRIPT_SEMANTIC_NODE_KINDS, JAVASCRIPT_SEMANTIC_RELATION_FAMILIES, JAVASCRIPT_SEMANTIC_RELATIONS, @@ -27,6 +28,7 @@ import { parseJavaScriptSemanticGraph, serializeJavaScriptSemanticGraph, } from "./javascriptSemanticGraphSerialization.js"; +import { queryJavaScriptSemanticGraph } from "./javascriptSemanticQuery.js"; const SHA = "a".repeat(64); const JAG_ID = `jag_${"b".repeat(64)}`; @@ -140,6 +142,44 @@ it("accepts semantic module paths longer than the former schema ceiling", () => ).toBe(modulePath); }); +it("captures caller-owned node identity, properties, and provenance", () => { + const contexts = new JavaScriptSemanticEvidenceContextRegistry(); + const input = { + kind: "literal", + identity: { + artifact_sha256: SHA, + module_path: "bundle.js", + source_range: null, + role_key: "literal", + }, + function_node_id: null, + application_node_ids: [], + label: "literal", + properties: { nested: ["original"] }, + evidence: evidence(), + }; + const captured = createJavaScriptSemanticGraphNode(input, contexts); + const expected = structuredClone(captured); + const expectedEvidence = structuredClone(input.evidence); + + input.identity.module_path = "replaced.js"; + input.properties.nested[0] = "replaced"; + input.evidence.limitations.push("later mutation"); + if (input.evidence.location.available) + input.evidence.location.value = { + kind: "artifact-path", + path: "replaced.js", + }; + + expect(captured).toEqual(expected); + expect( + resolveJavaScriptSemanticEvidence( + { evidence_contexts: contexts.contexts }, + captured.evidence, + ), + ).toEqual(expectedEvidence); +}); + const fixtureInput = (withUnknown = false): JavaScriptSemanticGraphInput => { const evidenceContexts = new JavaScriptSemanticEvidenceContextRegistry(); const module = node("module", "module", {}, evidenceContexts); @@ -285,6 +325,104 @@ const registryForGraph = ( return registry; }; +const graphWithCandidateRelations = (): JavaScriptSemanticGraph => { + const base = fixtureGraph(); + const evidenceContexts = registryForGraph(base); + const relations = base.relations.map( + ({ relation_id: _relationId, ...relation }) => + createJavaScriptSemanticGraphRelation( + { + ...relation, + resolution: "candidate", + evidence: resolveJavaScriptSemanticEvidence(base, relation.evidence), + }, + evidenceContexts, + ), + ); + const binding = base.nodes.find(({ label }) => label === "binding"); + const callable = base.nodes.find(({ label }) => label === "function"); + if (binding === undefined || callable === undefined) + throw new Error("Expected candidate-cycle endpoints"); + for (const [source, target] of [ + [binding, callable], + [callable, binding], + ] as const) + relations.push( + createJavaScriptSemanticGraphRelation( + { + source_node_id: source.node_id, + target_node_id: target.node_id, + relation: "owns", + resolution: "candidate", + properties: {}, + evidence: evidence("inferred"), + }, + evidenceContexts, + ), + ); + + const { graph_id: _graphId, ...input } = base; + return createJavaScriptSemanticGraph({ + ...input, + evidence_contexts: evidenceContexts.contexts, + relations, + coverage: { + ...input.coverage, + families: input.coverage.families.map((family) => ({ + ...family, + retained_relations: relations.filter( + ({ relation }) => + JAVASCRIPT_SEMANTIC_RELATION_FAMILY[relation] === family.family, + ).length, + })), + }, + }); +}; + +it("counts candidate edges by query direction and relation filter", () => { + const graph = graphWithCandidateRelations(); + const binding = graph.nodes.find(({ label }) => label === "binding"); + if (binding === undefined) throw new Error("Expected binding seed"); + const query = ( + direction: "forward-influence" | "backward-provenance", + allowedRelations: (typeof JAVASCRIPT_SEMANTIC_RELATIONS)[number][], + ) => + queryJavaScriptSemanticGraph(graph, { + seed: { kind: "semantic-node", node_id: binding.node_id }, + direction, + allowed_relations: allowedRelations, + }); + + expect(query("forward-influence", ["captures"]).status).toBe("ambiguous"); + expect(query("forward-influence", ["defines"]).status).toBe("found"); + expect(query("backward-provenance", ["defines"]).status).toBe("ambiguous"); + expect(query("backward-provenance", ["captures"]).status).toBe("found"); +}); + +it("counts candidate ownership cycles once and traverses them finitely", () => { + const graph = graphWithCandidateRelations(); + const binding = graph.nodes.find(({ label }) => label === "binding"); + if (binding === undefined) throw new Error("Expected binding seed"); + const result = queryJavaScriptSemanticGraph(graph, { + seed: { kind: "semantic-node", node_id: binding.node_id }, + direction: "ownership", + allowed_relations: ["owns"], + include_ambiguous_dynamic_edges: true, + }); + + expect(result.status).toBe("ambiguous"); + expect(result.relations).toHaveLength(2); + expect(result.summary.traversed_nodes).toBe(2); + expect(result.summary.traversed_relations).toBe(2); + expect( + queryJavaScriptSemanticGraph(graph, { + seed: { kind: "semantic-node", node_id: binding.node_id }, + direction: "ownership", + allowed_relations: ["constructs-request"], + }).status, + ).toBe("found"); +}); + it("defines every required relation family and canonicalizes records", () => { expect(JAVASCRIPT_SEMANTIC_RELATIONS).toContain("argument-to-parameter"); expect(JAVASCRIPT_SEMANTIC_RELATIONS).toContain("detaches-task"); diff --git a/src/domain/javascript/javascriptSemanticGraphTraversal.test.ts b/src/domain/javascript/javascriptSemanticGraphTraversal.test.ts index e42af295f1..b73a83ee39 100644 --- a/src/domain/javascript/javascriptSemanticGraphTraversal.test.ts +++ b/src/domain/javascript/javascriptSemanticGraphTraversal.test.ts @@ -316,6 +316,20 @@ it("returns every relevant unknown inline", () => { expect(result.unknowns).toHaveLength(1); }); +it("does not admit a semantic-node seed absent from the graph", () => { + const result = queryJavaScriptSemanticGraph(fixtureGraph(), { + seed: { kind: "semantic-node", node_id: `jsrg_node_${"f".repeat(64)}` }, + direction: "forward-influence", + }); + + expect(result).toMatchObject({ + status: "no-match", + seed_node_ids: [], + nodes: [], + summary: { total_seed_matches: 0, traversed_nodes: 0 }, + }); +}); + it("returns the complete deterministic forward influence result inline", () => { const graph = fixtureGraph(); const input = { diff --git a/src/domain/javascript/javascriptSemanticMemberMutations.ts b/src/domain/javascript/javascriptSemanticMemberMutations.ts index 12ceec71d0..ed5d2fb090 100644 --- a/src/domain/javascript/javascriptSemanticMemberMutations.ts +++ b/src/domain/javascript/javascriptSemanticMemberMutations.ts @@ -40,11 +40,6 @@ export function* collectSemanticMemberMutationsSteps( state: JavaScriptSemanticAnalysisState, ): Generator { const parents = state.parentsByNode; - yield* traverseJavaScriptAstSteps(program, { - enter: (node, parent) => { - if (parent !== null) parents.set(node, parent); - }, - }); const normalizePath = observableMutationPaths(state); const returnedReferences = createSemanticReturnedReferences(state); const recordedEffects = new Set(); @@ -670,10 +665,10 @@ type MutationObservableOrigin = | "opaque"; interface MutationObserverNode { - readonly origins: Set; - readonly sources: Set; - readonly dependents: Set; - readonly projections: Map; + origins?: Set; + sources?: Set; + dependents?: Set; + projections?: Map; readonly read?: { readonly source: MutationObserverNode; readonly key: PropertyPath[number]; @@ -700,22 +695,17 @@ const observableMutationPaths = ( queued.add(node); pending.push(node); }; - const create = ( - read?: MutationObserverNode["read"], - ): MutationObserverNode => ({ - origins: new Set(), - sources: new Set(), - dependents: new Set(), - projections: new Map(), - ...(read === undefined ? {} : { read }), - }); + const create = (read?: MutationObserverNode["read"]): MutationObserverNode => + // Most bindings have no object origin or property projection. Materialize + // each collection only when it contains a fact or relationship. + read === undefined ? {} : { read }; const link = ( source: MutationObserverNode, target: MutationObserverNode, ): void => { - if (target.sources.has(source)) return; - target.sources.add(source); - source.dependents.add(target); + if (target.sources?.has(source) === true) return; + (target.sources ??= new Set()).add(source); + (source.dependents ??= new Set()).add(target); schedule(target); }; const expressionNode = (node: t.Node): MutationObserverNode => { @@ -736,11 +726,11 @@ const observableMutationPaths = ( const identity = JSON.stringify( key === null || typeof key === "object" ? key : String(key), ); - let selected = observer.projections.get(identity); + let selected = observer.projections?.get(identity); if (selected === undefined) { selected = create({ source: observer, key }); - observer.projections.set(identity, selected); - observer.dependents.add(selected); + (observer.projections ??= new Map()).set(identity, selected); + (observer.dependents ??= new Set()).add(selected); schedule(selected); } observer = selected; @@ -760,7 +750,7 @@ const observableMutationPaths = ( // Preserve exact paths where copies, defaults or iteration need their // reference-specific projections; an opaque origin prevents pruning. if (binding.referenceInitializers.length > 0) - observer.origins.add("opaque"); + (observer.origins ??= new Set()).add("opaque"); for (const initializer of binding.referenceInitializers) { for (const source of [ initializer, @@ -777,7 +767,7 @@ const observableMutationPaths = ( if (selected === undefined) break; const { expression, observer } = selected; if (t.isObjectExpression(expression) || t.isArrayExpression(expression)) - observer.origins.add(expression); + (observer.origins ??= new Set()).add(expression); else if (t.isIdentifier(expression)) { const binding = resolveSemanticBindingState( state, @@ -787,11 +777,12 @@ const observableMutationPaths = ( if (binding !== undefined) { const origin = bindingNodes.get(binding.bindingId); if (origin !== undefined) link(origin, observer); - } else if (expression.name === "Array") observer.origins.add("array"); + } else if (expression.name === "Array") + (observer.origins ??= new Set()).add("array"); else if ( ["globalThis", "global", "window", "self"].includes(expression.name) ) - observer.origins.add("ambient"); + (observer.origins ??= new Set()).add("ambient"); } else if ( t.isMemberExpression(expression) || t.isOptionalMemberExpression(expression) @@ -818,21 +809,21 @@ const observableMutationPaths = ( queued.delete(observer); let changed = false; const add = (origin: MutationObservableOrigin): void => { - if (observer.origins.has(origin)) return; - observer.origins.add(origin); + if (observer.origins?.has(origin) === true) return; + (observer.origins ??= new Set()).add(origin); changed = true; }; - for (const source of observer.sources) - for (const origin of source.origins) add(origin); + for (const source of observer.sources ?? []) + for (const origin of source.origins ?? []) add(origin); if (observer.read !== undefined) { const { source, key } = observer.read; if (key === null || typeof key === "object") { // Dynamic keys and copied-slot masks retain their original paths. // Enumerating their exclusion combinations would recreate the growth // that the mutation collector's selection union already prevents. - if (source.origins.size > 0) add("opaque"); + if ((source.origins?.size ?? 0) > 0) add("opaque"); } else - for (const origin of source.origins) { + for (const origin of source.origins ?? []) { if (typeof origin === "string") { if (origin === "array-prototype" || origin === "opaque") add(origin); @@ -869,7 +860,7 @@ const observableMutationPaths = ( } } if (changed) - for (const dependent of observer.dependents) schedule(dependent); + for (const dependent of observer.dependents ?? []) schedule(dependent); } }; settle(); @@ -881,15 +872,16 @@ const observableMutationPaths = ( if (effect === "escape") { observer = project(observer, path); settle(); - return observer.origins.size === 0 ? null : path; + return (observer.origins?.size ?? 0) === 0 ? null : path; } - if (observer.origins.size === 0) return path.length === 0 ? path : [null]; + if ((observer.origins?.size ?? 0) === 0) + return path.length === 0 ? path : [null]; for (let offset = 0; offset < path.length; offset++) { const key = path[offset]; if (key === undefined) break; observer = project(observer, [key]); settle(); - if (observer.origins.size === 0 && offset + 1 < path.length) + if ((observer.origins?.size ?? 0) === 0 && offset + 1 < path.length) return [...path.slice(0, offset + 1), null]; } return path; diff --git a/src/domain/javascript/javascriptSemanticObjects.ts b/src/domain/javascript/javascriptSemanticObjects.ts index 27f5d73b3c..08acf25f8a 100644 --- a/src/domain/javascript/javascriptSemanticObjects.ts +++ b/src/domain/javascript/javascriptSemanticObjects.ts @@ -179,7 +179,13 @@ const addObjectOperation = ( output: JavaScriptSemanticObjectOperation[], ): void => { output.push({ - objectOperationId: `object:${input.kind}:${String(input.node.start ?? -1)}:${String(input.node.end ?? -1)}`, + // These high-volume IDs must not retain intermediate concatenations. + objectOperationId: [ + "object", + input.kind, + String(input.node.start ?? -1), + String(input.node.end ?? -1), + ].join(":"), kind: input.kind, location: range(input.node), ownerCallableId: input.ownerCallableId, diff --git a/src/domain/javascript/javascriptSemanticQuery.ts b/src/domain/javascript/javascriptSemanticQuery.ts index 6439d8954f..a101cd934c 100644 --- a/src/domain/javascript/javascriptSemanticQuery.ts +++ b/src/domain/javascript/javascriptSemanticQuery.ts @@ -3,7 +3,6 @@ import type { JavaScriptSemanticGraphNode, JavaScriptSemanticGraphRelation, } from "./javascriptSemanticGraphSchemas.js"; -import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js"; import type { JavaScriptSemanticGraphUnknown } from "./javascriptSemanticGraphSchemas.js"; import { isJavaScriptSemanticOwnershipRelation as ownershipRelation } from "./javascriptSemanticQueryRelations.js"; import { @@ -30,10 +29,6 @@ interface Traversal { readonly modules: Set; } -interface SeedAdmission { - readonly nodeIds: string[]; -} - interface QueryResult { readonly retainedNodes: JavaScriptSemanticGraphNode[]; readonly retainedRelations: JavaScriptSemanticGraphRelation[]; @@ -47,17 +42,15 @@ export const queryJavaScriptSemanticGraph = ( const input = javaScriptSemanticQueryInputSchema.parse(rawInput); const queryId = queryIdentifier(graph, input); const seeds = resolveSeeds(graph, input); - const admission = admitSeeds(graph, seeds); - const retainedSeeds = admission.nodeIds; + const nodes = new Map(graph.nodes.map((node) => [node.node_id, node])); const adjacency = buildAdjacency(graph.relations, input); - const traversal = traverse(graph, retainedSeeds, adjacency); + const traversal = traverse(nodes, seeds, adjacency); const result = createQueryResult(graph, traversal); - const allRelevantUnknowns = relevantUnknownFrontiers( + const relevantUnknowns = relevantUnknownFrontiers( graph, traversal.nodeIds, input, ); - const relevantUnknowns = allRelevantUnknowns; const candidateRelations = relevantCandidateRelationCount( graph, traversal.nodeIds, @@ -87,7 +80,7 @@ export const queryJavaScriptSemanticGraph = ( seed: input.seed, direction: input.direction, status: assessment.status, - seed_node_ids: retainedSeeds, + seed_node_ids: seeds, evidence_contexts: graph.evidence_contexts.filter(({ context_id }) => retainedContextIds.has(context_id), ), @@ -101,7 +94,7 @@ export const queryJavaScriptSemanticGraph = ( traversed_relations: traversal.relationIds.size, traversed_functions: traversal.functionIds.size, traversed_modules: traversal.modules.size, - relevant_unknowns: allRelevantUnknowns.length, + relevant_unknowns: relevantUnknowns.length, }, coverage: assessment.coverage, limitations: assessment.limitations, @@ -121,20 +114,6 @@ const createQueryResult = ( return { retainedNodes, retainedRelations }; }; -const admitSeeds = ( - graph: JavaScriptSemanticGraph, - seeds: readonly string[], -): SeedAdmission => { - const nodes = new Map(graph.nodes.map((node) => [node.node_id, node])); - const retained: string[] = []; - for (const nodeId of seeds) { - const node = nodes.get(nodeId); - if (node === undefined) continue; - retained.push(nodeId); - } - return { nodeIds: retained }; -}; - const buildAdjacency = ( relations: readonly JavaScriptSemanticGraphRelation[], input: JavaScriptSemanticQueryInput, @@ -153,13 +132,6 @@ const buildAdjacency = ( continue; addDirectedEntries(adjacency, relation, input.direction); } - for (const entries of adjacency.values()) - entries.sort((left, right) => - compareUnicodeCodePoints( - `${left.relation.relation_id}\0${left.nextNodeId}`, - `${right.relation.relation_id}\0${right.nextNodeId}`, - ), - ); return adjacency; }; @@ -222,11 +194,10 @@ const addEntry = ( }; const traverse = ( - graph: JavaScriptSemanticGraph, + nodes: ReadonlyMap, seeds: readonly string[], adjacency: ReadonlyMap, ): Traversal => { - const nodes = new Map(graph.nodes.map((node) => [node.node_id, node])); const nodeIds = new Set(seeds); const relationIds = new Set(); const functionIds = new Set(); @@ -289,15 +260,32 @@ const relevantCandidateRelationCount = ( nodeIds: ReadonlySet, input: JavaScriptSemanticQueryInput, ): number => { - const adjacency = buildAdjacency( - graph.relations.filter(({ resolution }) => resolution === "candidate"), - { ...input, include_ambiguous_dynamic_edges: true }, - ); - const relevant = new Set(); - for (const nodeId of nodeIds) - for (const { relation } of adjacency.get(nodeId) ?? []) - relevant.add(relation.relation_id); - return relevant.size; + const allowed = + input.allowed_relations === undefined + ? null + : new Set(input.allowed_relations); + let relevant = 0; + for (const relation of graph.relations) { + if ( + relation.resolution !== "candidate" || + (allowed !== null && !allowed.has(relation.relation)) + ) + continue; + const sourceIsRelevant = nodeIds.has(relation.source_node_id); + const targetIsRelevant = nodeIds.has(relation.target_node_id); + if ( + (input.direction === "callers" && + relation.relation === "calls" && + targetIsRelevant) || + (input.direction === "ownership" && + ownershipRelation(relation.relation) && + (sourceIsRelevant || targetIsRelevant)) || + (input.direction === "backward-provenance" && targetIsRelevant) || + (input.direction === "forward-influence" && sourceIsRelevant) + ) + relevant += 1; + } + return relevant; }; const expectedMatchesFor = ( diff --git a/src/domain/javascript/javascriptSemanticReturns.ts b/src/domain/javascript/javascriptSemanticReturns.ts index fb51279bf0..80aeb1eb36 100644 --- a/src/domain/javascript/javascriptSemanticReturns.ts +++ b/src/domain/javascript/javascriptSemanticReturns.ts @@ -83,7 +83,14 @@ const semanticReturnSiteId = ( callableId: string, location: JavaScriptSemanticReturnSite["location"], ): string => - `${callableId}:return:${String(location.start.line)}:${String(location.start.column)}:${String(location.end.line)}:${String(location.end.column)}`; + [ + callableId, + "return", + String(location.start.line), + String(location.start.column), + String(location.end.line), + String(location.end.column), + ].join(":"); /** Link exports to callables only when lexical resolution is unique and exact. */ export const resolveSemanticModuleCallables = ( diff --git a/src/domain/javascript/javascriptSemanticState.ts b/src/domain/javascript/javascriptSemanticState.ts index 28ca97371e..f7ef253196 100644 --- a/src/domain/javascript/javascriptSemanticState.ts +++ b/src/domain/javascript/javascriptSemanticState.ts @@ -60,6 +60,7 @@ export interface JavaScriptSemanticAnalysisState { readonly parentsByNode: WeakMap; readonly scopes: JavaScriptSemanticScopeState[]; readonly scopesById: Map; + /** Scope boundaries and descendant scopes resolved on demand. */ readonly scopeByNode: WeakMap; readonly bindingsById: Map; readonly callables: JavaScriptSemanticCallable[]; @@ -94,7 +95,7 @@ export const resolveSemanticBindingState = ( node: t.Node, name: string, ): JavaScriptSemanticBindingState | undefined => { - let scope = state.scopeByNode.get(node); + let scope = semanticScopeForNode(state, node); while (scope !== undefined) { const binding = scope.bindings.get(name); if (binding !== undefined) return binding; @@ -113,7 +114,7 @@ export const semanticResolutionBlocked = ( node: t.Node, name: string, ): boolean => { - let scope = state.scopeByNode.get(node); + let scope = semanticScopeForNode(state, node); while (scope !== undefined) { if (scope.bindings.has(name)) return false; if (!scope.bindingsComplete) return true; @@ -125,6 +126,28 @@ export const semanticResolutionBlocked = ( return false; }; +const semanticScopeForNode = ( + state: JavaScriptSemanticAnalysisState, + node: t.Node, +): JavaScriptSemanticScopeState | undefined => { + let candidate: t.Node | undefined = node; + while (candidate !== undefined) { + const scope = state.scopeByNode.get(candidate); + if (scope !== undefined) { + // Repeated references in a deep expression share this ancestry. Cache + // the traversed path without eagerly indexing nodes never resolved. + let descendant: t.Node | undefined = node; + while (descendant !== undefined && descendant !== candidate) { + state.scopeByNode.set(descendant, scope); + descendant = state.parentsByNode.get(descendant); + } + return scope; + } + candidate = state.parentsByNode.get(candidate); + } + return undefined; +}; + /** Resolve one name starting from an explicitly selected lexical scope. */ export const resolveSemanticBindingFromScope = ( scope: JavaScriptSemanticScopeState, diff --git a/src/domain/jsonShape.ts b/src/domain/jsonShape.ts index 2976d3ceb4..1772585d19 100644 --- a/src/domain/jsonShape.ts +++ b/src/domain/jsonShape.ts @@ -33,6 +33,15 @@ export const jsonShapeSchema = z.object({ }); export type JsonShape = z.infer; +const arrayElementKey = Symbol("array-element"); + +interface ShapeNode { + readonly path: JsonShapePathSegment[]; + readonly types: Set; + observations: number; + children?: Map; +} + /** Parse approved JSON and immediately discard values after iterative shape inference. */ export const inferJsonShape = (text: string): JsonShape | null => { let root: unknown; @@ -43,48 +52,52 @@ export const inferJsonShape = (text: string): JsonShape | null => { void cause; return null; } - const properties = new Map< - string, - { - readonly path: JsonShapePathSegment[]; - readonly types: Set; - observations: number; - } - >(); + const properties: ShapeNode[] = []; + const rootShape: ShapeNode = { + path: [], + types: new Set(), + observations: 0, + }; + // Each structural path owns one accumulator, shared by all matching array rows. + const childShape = ( + parent: ShapeNode, + key: string | typeof arrayElementKey, + ): ShapeNode => { + const children = (parent.children ??= new Map()); + const existing = children.get(key); + if (existing !== undefined) return existing; + const node: ShapeNode = { + path: [ + ...parent.path, + typeof key === "string" + ? { kind: "property", name: key } + : { kind: "array-element" }, + ], + types: new Set(), + observations: 0, + }; + children.set(key, node); + properties.push(node); + return node; + }; const pending: Array<{ readonly value: unknown; - readonly path: JsonShapePathSegment[]; - readonly depth: number; - }> = [{ value: root, path: [], depth: 0 }]; + readonly shape: ShapeNode; + }> = [{ value: root, shape: rootShape }]; let nodeCount = 0; let maxDepthObserved = 0; while (pending.length > 0) { const current = pending.pop(); if (current === undefined) break; nodeCount += 1; - maxDepthObserved = Math.max(maxDepthObserved, current.depth); - if (current.path.length > 0) { - const key = JSON.stringify(current.path); - const type = jsonValueType(current.value); - const existing = properties.get(key); - if (existing === undefined) - properties.set(key, { - path: current.path, - types: new Set([type]), - observations: 1, - }); - else { - existing.types.add(type); - existing.observations += 1; - } - } + maxDepthObserved = Math.max(maxDepthObserved, current.shape.path.length); + current.shape.types.add(jsonValueType(current.value)); + current.shape.observations += 1; if (Array.isArray(current.value)) { + if (current.value.length === 0) continue; + const shape = childShape(current.shape, arrayElementKey); for (let index = current.value.length - 1; index >= 0; index -= 1) - pending.push({ - value: current.value[index], - path: [...current.path, { kind: "array-element" }], - depth: current.depth + 1, - }); + pending.push({ value: current.value[index], shape }); continue; } if (!isRecord(current.value)) continue; @@ -93,18 +106,14 @@ export const inferJsonShape = (text: string): JsonShape | null => { const entry = allEntries[index]; if (entry === undefined) continue; const [name, value] = entry; - const path: JsonShapePathSegment[] = [ - ...current.path, - { kind: "property", name }, - ]; - pending.push({ value, path, depth: current.depth + 1 }); + pending.push({ value, shape: childShape(current.shape, name) }); } } return jsonShapeSchema.parse({ root_type: jsonValueType(root), node_count: nodeCount, max_depth_observed: maxDepthObserved, - properties: [...properties.values()] + properties: properties .map((value) => ({ path: value.path, types: [...value.types].sort(), diff --git a/src/domain/managed/managedApplicationGraph.ts b/src/domain/managed/managedApplicationGraph.ts index f99ebb47cb..3e4ffea377 100644 --- a/src/domain/managed/managedApplicationGraph.ts +++ b/src/domain/managed/managedApplicationGraph.ts @@ -134,6 +134,7 @@ export const projectManagedApplicationGraph = ( const parsed = parseManagedInputs(input); const artifactIdentity = chooseArtifact(parsed); assertSameArtifact(parsed, artifactIdentity.sha256); + assertConsistentModuleIdentity(parsed); const evidenceLinks = [ parsed.artifact?.evidence.evidence_id, parsed.members?.evidence.evidence_id, @@ -258,6 +259,52 @@ const assertSameArtifact = ( ); }; +/** Reject contradictory known module identities while leaving null fields unknown. */ +const assertConsistentModuleIdentity = ( + parsed: ParsedManagedGraphInput, +): void => { + const mvidClaims = [ + parsed.artifact?.result.module?.mvid, + parsed.members?.result.module?.mvid, + parsed.members?.result.identity_scope.requires_mvid, + parsed.boundaries?.result.module?.mvid, + parsed.boundaries?.result.identity_scope.requires_mvid, + ] + .filter((mvid): mvid is string => mvid !== undefined && mvid !== null) + .map((mvid) => mvid.toLowerCase()); + if (new Set(mvidClaims).size > 1) + throw new TypeError( + "Managed graph inputs report conflicting known module MVIDs or token scopes", + ); + + const modules = [ + parsed.artifact?.result.module, + parsed.members?.result.module, + parsed.boundaries?.result.module, + ].filter( + (module): module is NonNullable => + module !== undefined && module !== null, + ); + for (const field of ["name", "generation", "token", "row_offset"] as const) { + const values = new Set(modules.map((module) => module[field])); + if (values.size > 1) + throw new TypeError( + `Managed graph inputs report conflicting module ${field} values`, + ); + } + for (const field of ["enc_id", "enc_base_id"] as const) { + const values = new Set( + modules + .map((module) => module[field]?.toLowerCase()) + .filter((value): value is string => value !== undefined), + ); + if (values.size > 1) + throw new TypeError( + `Managed graph inputs report conflicting module ${field} values`, + ); + } +}; + const omittedCounts = (parsed: ParsedManagedGraphInput) => assessManagedGraphOmissions({ artifact: parsed.artifact?.result ?? null, @@ -288,4 +335,9 @@ const projectionLimitations = ( ...(omissions.partialInput ? ["At least one supplied managed Evidence record has partial coverage."] : []), + ...(omissions.unscopedTokenRelationships + ? [ + "Managed boundary tokens were not linked to member nodes because matching module MVID scope was not established.", + ] + : []), ]; diff --git a/src/domain/managed/managedApplicationGraphCoverage.ts b/src/domain/managed/managedApplicationGraphCoverage.ts index ebdc79c6a5..dc15a64623 100644 --- a/src/domain/managed/managedApplicationGraphCoverage.ts +++ b/src/domain/managed/managedApplicationGraphCoverage.ts @@ -4,6 +4,7 @@ import type { ManagedMemberInspection, ManagedNativeBoundaryInspection, } from "./managedArtifact.js"; +import { managedTokenScopesMatch } from "./managedInspectionEvidence.js"; type ManagedCoverageState = "complete" | "partial" | "unavailable"; @@ -17,6 +18,7 @@ export interface ManagedGraphCoverageSources { /** Whether any supplied source has incomplete coverage. */ export interface ManagedGraphProjectionOmissions { readonly partialInput: boolean; + readonly unscopedTokenRelationships: boolean; } const completeCoverage = (): ApplicationGraphEvidence["coverage"] => ({ @@ -43,6 +45,17 @@ export const managedSourceCoverage = ( export const assessManagedGraphOmissions = ( sources: ManagedGraphCoverageSources, ): ManagedGraphProjectionOmissions => ({ + unscopedTokenRelationships: + sources.members !== null && + sources.boundaries !== null && + (sources.boundaries.pinvoke_imports.some( + ({ member_token }) => member_token !== null, + ) || + sources.boundaries.native_implementations.length > 0) && + !managedTokenScopesMatch( + sources.members.identity_scope.requires_mvid, + sources.boundaries.identity_scope.requires_mvid, + ), partialInput: [ sources.members?.coverage.state, sources.boundaries?.coverage.state, @@ -54,7 +67,7 @@ export const assessManagedGraphOmissions = ( export const managedGraphEvidenceCoverage = ( omissions: ManagedGraphProjectionOmissions, ): ApplicationGraphEvidence["coverage"] => - omissions.partialInput + omissions.partialInput || omissions.unscopedTokenRelationships ? { status: "partial", truncated: false, @@ -67,7 +80,8 @@ export const managedGraphEvidenceCoverage = ( export const managedGraphResultCoverage = ( omissions: ManagedGraphProjectionOmissions, ) => ({ - status: omissions.partialInput - ? ("partial" as const) - : ("complete-within-inputs" as const), + status: + omissions.partialInput || omissions.unscopedTokenRelationships + ? ("partial" as const) + : ("complete-within-inputs" as const), }); diff --git a/src/domain/managed/managedApplicationGraphNodes.ts b/src/domain/managed/managedApplicationGraphNodes.ts index cd631d018a..ff186cecc9 100644 --- a/src/domain/managed/managedApplicationGraphNodes.ts +++ b/src/domain/managed/managedApplicationGraphNodes.ts @@ -7,6 +7,7 @@ import type { ApplicationGraphEvidence } from "../javascript/javascriptApplicati import type { ApplicationNode } from "../javascript/javascriptApplicationGraphSchemas.js"; import { managedSourceCoverage } from "./managedApplicationGraphCoverage.js"; import type { Evidence } from "../evidence.js"; +import { managedTokenScopesMatch } from "./managedInspectionEvidence.js"; import { type ManagedArtifactInspection, type ManagedMemberInspection, @@ -106,7 +107,7 @@ export const addArtifactIdentityNodes = ( identity: artifactLocalIdentity( state.artifactSha256, "managed-module", - module.mvid ?? module.token, + module.mvid?.toLowerCase() ?? module.token, ), observations: [ { @@ -138,25 +139,32 @@ interface ManagedModuleSource { const selectModuleSource = ( parsed: ParsedManagedGraphInput, ): ManagedModuleSource | null => { - if (parsed.artifact?.result.module != null) - return { - module: parsed.artifact.result.module, - evidence: parsed.artifact.evidence, - coverageState: parsed.artifact.result.coverage.state, - }; - if (parsed.members?.result.module != null) - return { - module: parsed.members.result.module, - evidence: parsed.members.evidence, - coverageState: parsed.members.result.coverage.state, - }; - if (parsed.boundaries?.result.module != null) - return { - module: parsed.boundaries.result.module, - evidence: parsed.boundaries.evidence, - coverageState: parsed.boundaries.result.coverage.state, - }; - return null; + const sources = [ + parsed.artifact?.result.module == null + ? null + : { + module: parsed.artifact.result.module, + evidence: parsed.artifact.evidence, + coverageState: parsed.artifact.result.coverage.state, + }, + parsed.members?.result.module == null + ? null + : { + module: parsed.members.result.module, + evidence: parsed.members.evidence, + coverageState: parsed.members.result.coverage.state, + }, + parsed.boundaries?.result.module == null + ? null + : { + module: parsed.boundaries.result.module, + evidence: parsed.boundaries.evidence, + coverageState: parsed.boundaries.result.coverage.state, + }, + ].filter((source): source is ManagedModuleSource => source !== null); + return ( + sources.find(({ module }) => module.mvid !== null) ?? sources[0] ?? null + ); }; /** Add managed type, method, and field nodes from member evidence. */ @@ -332,11 +340,17 @@ export const addBoundaryNodes = ( boundaryInput.evidence, boundaries.coverage.state, ); + const canJoinMemberTokens = + parsed.members !== null && + managedTokenScopesMatch( + parsed.members.result.identity_scope.requires_mvid, + boundaries.identity_scope.requires_mvid, + ); for (const pinvoke of pinvokes) { const node = pinvokeNode(state, pinvoke, sourceEvidence); state.nodes.push(node); const method = - pinvoke.member_token === null + pinvoke.member_token === null || !canJoinMemberTokens ? undefined : state.methodNodes.get(pinvoke.member_token); const memberEvidence = @@ -370,7 +384,10 @@ export const addBoundaryNodes = ( sourceEvidence, ); state.nodes.push(node); - const owner = state.methodNodes.get(implementation.token) ?? artifactNode; + const owner = + (canJoinMemberTokens + ? state.methodNodes.get(implementation.token) + : undefined) ?? artifactNode; addContainsEdge(state, owner, node, { kind: "declares-managed-native-implementation", coverage: implementationCoverage, diff --git a/src/domain/managed/managedInspectionEvidence.ts b/src/domain/managed/managedInspectionEvidence.ts index 3bd35ea7b7..fa4712b5d0 100644 --- a/src/domain/managed/managedInspectionEvidence.ts +++ b/src/domain/managed/managedInspectionEvidence.ts @@ -1,12 +1,28 @@ import { parseEvidence, type Evidence } from "../evidence.js"; import type { z } from "zod"; +/** Build-local metadata tokens join only under a known, matching module MVID. */ +export const managedTokenScopesMatch = ( + leftMvid: string | null, + rightMvid: string | null, +): boolean => + leftMvid !== null && + rightMvid !== null && + leftMvid.toLowerCase() === rightMvid.toLowerCase(); + /** * Parse managed inspection Evidence and bind its artifact to any available subject. * A null subject remains valid and means there is no digest to cross-check. */ export const parseManagedInspectionEvidence = < - Result extends { readonly artifact: { readonly sha256: string } }, + Result extends { + readonly artifact: { readonly sha256: string }; + readonly identity_scope?: { + readonly requires_artifact_sha256: string; + readonly requires_mvid: string | null; + }; + readonly module?: { readonly mvid: string | null } | null; + }, >( rawEvidence: unknown, operation: string, @@ -22,5 +38,20 @@ export const parseManagedInspectionEvidence = < throw new TypeError( `Managed Evidence ${operation} (${evidence.evidence_id}) subject SHA-256 ${subjectSha256} does not match normalized artifact SHA-256 ${artifactSha256}`, ); + const scope = result.identity_scope; + if (scope !== undefined && scope.requires_artifact_sha256 !== artifactSha256) + throw new TypeError( + `Managed Evidence ${operation} (${evidence.evidence_id}) token scope SHA-256 ${scope.requires_artifact_sha256} does not match normalized artifact SHA-256 ${artifactSha256}`, + ); + if ( + scope?.requires_mvid !== undefined && + scope.requires_mvid !== null && + result.module?.mvid !== undefined && + result.module.mvid !== null && + scope.requires_mvid.toLowerCase() !== result.module.mvid.toLowerCase() + ) + throw new TypeError( + `Managed Evidence ${operation} (${evidence.evidence_id}) token scope MVID ${scope.requires_mvid} does not match module MVID ${result.module.mvid}`, + ); return { evidence, result }; }; diff --git a/src/domain/process/processScenario.test.ts b/src/domain/process/processScenario.test.ts new file mode 100644 index 0000000000..be7075502e --- /dev/null +++ b/src/domain/process/processScenario.test.ts @@ -0,0 +1,71 @@ +import { expect, it } from "vitest"; + +import { + parseProcessScenario, + PROCESS_TERMINAL_CELL_BUDGET, + processScenarioSchema, +} from "./processScenario.js"; + +it("admits the default and the exact combined terminal-cell budget", () => { + expect(parseProcessScenario({ executable: "/bin/true" }).terminal).toEqual({ + columns: 80, + rows: 24, + scrollback: 1_000, + }); + + expect( + processScenarioSchema.safeParse({ + executable: "/bin/true", + terminal: { + columns: 1, + rows: 1, + scrollback: PROCESS_TERMINAL_CELL_BUDGET - 1, + }, + }).success, + ).toBe(true); +}); + +it("rejects an oversized initial buffer without clamping caller dimensions", () => { + const input = { + executable: "/bin/true", + terminal: { columns: 1_000, rows: 1, scrollback: 1_000 }, + }; + const parsed = processScenarioSchema.safeParse(input); + + expect(parsed.success).toBe(false); + if (parsed.success) throw new Error("expected terminal capacity rejection"); + expect(parsed.error.issues).toContainEqual( + expect.objectContaining({ + code: "custom", + path: ["terminal"], + message: expect.stringContaining( + `${String(PROCESS_TERMINAL_CELL_BUDGET)} renderer cells`, + ), + }), + ); +}); + +it("applies the selected scrollback budget to every scheduled resize", () => { + const parsed = processScenarioSchema.safeParse({ + executable: "/bin/true", + events: [{ type: "resize", at_ms: 10, columns: 1_000, rows: 1 }], + }); + + expect(parsed.success).toBe(false); + if (parsed.success) throw new Error("expected resize capacity rejection"); + expect(parsed.error.issues).toContainEqual( + expect.objectContaining({ + code: "custom", + path: ["events", 0], + message: expect.stringContaining("resize event at 10 ms"), + }), + ); + + expect( + processScenarioSchema.safeParse({ + executable: "/bin/true", + terminal: { columns: 1, rows: 1, scrollback: 0 }, + events: [{ type: "resize", at_ms: 10, columns: 1_000, rows: 1 }], + }).success, + ).toBe(true); +}); diff --git a/src/domain/process/processScenario.ts b/src/domain/process/processScenario.ts index ff9197d158..c489db92e4 100644 --- a/src/domain/process/processScenario.ts +++ b/src/domain/process/processScenario.ts @@ -3,6 +3,12 @@ import canonicalize from "canonicalize"; import { z } from "zod"; const positiveBudget = z.number().int().safe().positive(); +/** + * Maximum xterm backing cells for one visible-plus-scrollback buffer. Each + * xterm BufferLine stores three Uint32 words per cell (12 MB at this cap), + * before row objects, serialization strings, and retained frame copies. + */ +export const PROCESS_TERMINAL_CELL_BUDGET = 1_000_000; const timedEventBase = { at_ms: z.number().int().safe().nonnegative() }; const reservedRunIdEnvironmentName = "REA_PROCESS_RUN_ID"; const ENVIRONMENT_NAME_CHARACTER = "[^=\\x00]"; @@ -41,6 +47,14 @@ const childProcessString = z /^[^\x00]*$/u, "Values passed to operating-system APIs cannot contain NUL", ); + +const fitsProcessTerminalCellBudget = ( + columns: number, + rows: number, + scrollback: number, +): boolean => + rows + scrollback <= Math.floor(PROCESS_TERMINAL_CELL_BUDGET / columns); + export const normalizationSchema = z.object({ paths: z.boolean(), pids: z.boolean(), @@ -131,8 +145,11 @@ export const processScenarioSchema = z .strictObject({ columns: z.number().int().min(1).max(65_535).default(80), rows: z.number().int().min(1).max(65_535).default(24), - scrollback: z.number().int().min(0).default(1_000), + scrollback: z.number().int().safe().nonnegative().default(1_000), }) + .describe( + "The visible terminal and scrollback together must use at most 1,000,000 xterm renderer cells: columns × (rows + scrollback). Resize events use the same scrollback and limit.", + ) .default({ columns: 80, rows: 24, scrollback: 1_000 }), events: z .array( @@ -204,6 +221,18 @@ export const processScenarioSchema = z }), }) .superRefine((scenario, context) => { + if ( + !fitsProcessTerminalCellBudget( + scenario.terminal.columns, + scenario.terminal.rows, + scenario.terminal.scrollback, + ) + ) + context.addIssue({ + code: "custom", + path: ["terminal"], + message: `terminal columns × (rows + scrollback) must not exceed ${String(PROCESS_TERMINAL_CELL_BUDGET)} renderer cells`, + }); for (let index = 1; index < scenario.events.length; index += 1) { const event = scenario.events[index]; const previous = scenario.events[index - 1]; @@ -215,7 +244,20 @@ export const processScenarioSchema = z }); } } - for (const event of scenario.events) { + for (const [index, event] of scenario.events.entries()) { + if ( + event.type === "resize" && + !fitsProcessTerminalCellBudget( + event.columns, + event.rows, + scenario.terminal.scrollback, + ) + ) + context.addIssue({ + code: "custom", + path: ["events", index], + message: `resize event at ${String(event.at_ms)} ms exceeds the ${String(PROCESS_TERMINAL_CELL_BUDGET)}-cell renderer budget with the selected scrollback`, + }); if (event.at_ms > scenario.timeout_ms) { context.addIssue({ code: "custom", diff --git a/src/evaluation/AgentEvaluationReport.ts b/src/evaluation/AgentEvaluationReport.ts index 37e6a70eb6..4af002f40c 100644 --- a/src/evaluation/AgentEvaluationReport.ts +++ b/src/evaluation/AgentEvaluationReport.ts @@ -3,6 +3,9 @@ export type FixtureCorrectness = "passed" | "failed" | "not_assessed"; /** Measurements consumed by the evaluation runner's release gate. */ export interface AgentEvaluationGateResult { + /** Explicit null requires a target clarification with no REA calls. */ + readonly expectedFirstTool?: string | null; + readonly targetClarificationPassed?: boolean; readonly exitCode: number; readonly naturalUse: boolean; readonly correctFirstTool: boolean; @@ -20,16 +23,20 @@ export const agentEvaluationPassed = ( result: AgentEvaluationGateResult, ): boolean => result.exitCode === 0 && - result.naturalUse && + (result.expectedFirstTool === null + ? !result.naturalUse + : result.naturalUse) && result.correctFirstTool && result.repeatedCallCount === 0 && result.inputValidationFailureCount === 0 && result.requiredToolSubsequenceMet && result.inputTokens > 0 && - (result.factualCorrectness === "passed" || - (result.factualCorrectness === "not_assessed" && - result.answerHeuristicsMet && - result.epistemicCuePresent)); + (result.expectedFirstTool === null + ? result.targetClarificationPassed === true + : result.factualCorrectness === "passed" || + (result.factualCorrectness === "not_assessed" && + result.answerHeuristicsMet && + result.epistemicCuePresent)); /** Keep assessed fixture outcomes and scenarios without truth oracles distinct. */ export const summarizeFactualCorrectness = ( diff --git a/src/evaluation/CodexAgentEval.ts b/src/evaluation/CodexAgentEval.ts index 2aebc7ca7d..ba5c500d05 100644 --- a/src/evaluation/CodexAgentEval.ts +++ b/src/evaluation/CodexAgentEval.ts @@ -4,6 +4,7 @@ import { type KnownAnswerAssessment, } from "./KnownAnswerEvaluation.js"; import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js"; +import { TOOL_CONTRACTS } from "../contracts/toolContracts.js"; /** One REA MCP invocation observed in a Codex JSONL transcript. */ interface CodexMcpCall { @@ -72,7 +73,35 @@ const evidenceIdsFrom = (value: unknown): readonly string[] => { return [...new Set(encoded.match(/ev_[a-f0-9]{64}/gu) ?? [])]; }; -const toolResultFailed = (item: Record): boolean => { +const toolResultError = ( + item: Record, +): Record | undefined => { + const result = record(item.result ?? item.output); + const structured = record( + result?.structured_content ?? result?.structuredContent, + ); + const direct = record(structured?.error ?? result?.error ?? item.error); + if (direct !== undefined || structured !== undefined) return direct; + // Codex preserves some REA failures only as one complete JSON text block. + // A client-truncated preview or SDK prose is not a structured error record. + const content = result?.content; + if (!Array.isArray(content) || content.length !== 1) return undefined; + const block = record(content[0]); + if (block?.type !== "text" || typeof block.text !== "string") + return undefined; + try { + return record(record(JSON.parse(block.text))?.error); + } catch (cause) { + if (cause instanceof SyntaxError || cause instanceof RangeError) + return undefined; + throw cause; + } +}; + +const toolResultFailed = ( + item: Record, + error: Record | undefined, +): boolean => { if ( item.status === "failed" || (item.error !== undefined && item.error !== null) @@ -82,16 +111,11 @@ const toolResultFailed = (item: Record): boolean => { const structured = record( result?.structured_content ?? result?.structuredContent, ); - return result?.isError === true || structured?.error !== undefined; -}; - -const toolResultErrorCode = (item: Record): string | null => { - const result = record(item.result ?? item.output); - const structured = record( - result?.structured_content ?? result?.structuredContent, + return ( + result?.isError === true || + structured?.error !== undefined || + error !== undefined ); - const error = record(structured?.error ?? result?.error ?? item.error); - return textValue(error?.code) ?? null; }; const callFromItem = (value: unknown): CodexMcpCall | undefined => { @@ -99,14 +123,15 @@ const callFromItem = (value: unknown): CodexMcpCall | undefined => { if (item?.type !== "mcp_tool_call") return undefined; const tool = textValue(item.tool) ?? textValue(item.name); if (tool === undefined) return undefined; + const error = toolResultError(item); return { id: textValue(item.id) ?? null, server: textValue(item.server) ?? textValue(item.server_name) ?? null, tool, arguments: item.arguments ?? item.input ?? {}, evidenceIds: evidenceIdsFrom(item.result ?? item.output), - error: toolResultFailed(item), - errorCode: toolResultErrorCode(item), + error: toolResultFailed(item, error), + errorCode: textValue(error?.code) ?? null, }; }; @@ -171,7 +196,7 @@ const consumeEvent = (state: EvaluationState, value: unknown): void => { /** Measure agent routing, repetition, model usage, and explicitly limited answer heuristics. */ export const evaluateCodexEvents = ( events: readonly unknown[], - expectedFirstTool: string, + expectedFirstTool: string | null, options: { readonly requireEvidence?: boolean; readonly requiredAnswerTermGroups?: readonly (readonly string[])[]; @@ -194,16 +219,31 @@ export const evaluateCodexEvents = ( ({ server }) => server === null || server.toLowerCase() === "rea", ); const signatures = new Map(); + let targetRevision = 0; for (const call of reaCalls) { - const signature = `${call.tool}:${JSON.stringify(canonicalValue(call.arguments))}`; + // A status read before and after a successful target lifecycle change is + // useful verification, not a duplicate analysis or unchanged failed retry. + const context = + call.tool === "binary_session" ? `${String(targetRevision)}:` : ""; + const signature = `${context}${call.tool}:${JSON.stringify(canonicalValue(call.arguments))}`; signatures.set(signature, (signatures.get(signature) ?? 0) + 1); + if ( + !call.error && + (call.tool === "open_binary" || call.tool === "close_binary") + ) + targetRevision += 1; } const repeatedCallCount = [...signatures.values()].reduce( (total, count) => total + Math.max(0, count - 1), 0, ); const inputValidationFailureCount = reaCalls.filter( - ({ errorCode }) => errorCode === "invalid_request", + (call) => + call.errorCode === "invalid_request" || + (call.error && + TOOL_CONTRACTS.find( + ({ name }) => name === call.tool, + )?.inputSchema.safeParse(call.arguments).success === false), ).length; const requiredToolSubsequenceMet = containsOrderedSubsequence( reaCalls.filter(({ error }) => !error).map(({ tool }) => tool), @@ -234,7 +274,7 @@ export const evaluateCodexEvents = ( ); return { naturalUse: reaCalls.length > 0, - correctFirstTool: reaCalls[0]?.tool === expectedFirstTool, + correctFirstTool: (reaCalls[0]?.tool ?? null) === expectedFirstTool, firstTool: reaCalls[0]?.tool ?? null, reaCalls, repeatedCallCount, diff --git a/src/evm/EvmoleFailures.ts b/src/evm/EvmoleFailures.ts index 5690a079ab..2677a3e719 100644 --- a/src/evm/EvmoleFailures.ts +++ b/src/evm/EvmoleFailures.ts @@ -59,8 +59,6 @@ export const evmInterfaceFailure = ( message: cause.message, exit_code: cause.snapshot?.exitCode ?? null, signal: cause.snapshot?.signal ?? null, - stdout: cause.snapshot?.stdout.text ?? null, - stderr: cause.snapshot?.stderr.text ?? null, }, }, { operation: "inspect_evm_interface", cause }, diff --git a/src/evm/EvmoleInterfaceProvider.ts b/src/evm/EvmoleInterfaceProvider.ts index 42a4e08480..03731a92a6 100644 --- a/src/evm/EvmoleInterfaceProvider.ts +++ b/src/evm/EvmoleInterfaceProvider.ts @@ -401,28 +401,8 @@ export class EvmoleInterfaceProvider implements EvmInterfacePort { } if (processOwner !== undefined) { this.#pendingCleanup = { root, process: processOwner }; - root = undefined; - return err( - new ProviderCleanupError( - EVMOLE_PROVIDER_IDENTITY.id, - [ - processOwner.launch.ownership?.runId ?? "owned-worker", - ...(this.#pendingCleanup.root === undefined - ? [] - : [this.#pendingCleanup.root.path]), - ], - { - reason: "Owned EVM worker cleanup could not be confirmed", - previous_error: result.ok - ? null - : projectAnalysisError(result.error), - ...(retainedOutput === undefined - ? {} - : { captured_output: { ...retainedOutput } }), - }, - { operation: OPERATION }, - ), - ); + // The command failure already owns this cleanup diagnostic and its resources. + return result; } if (root !== undefined) { try { diff --git a/src/firmware/FirmwareProvider.test.ts b/src/firmware/FirmwareProvider.test.ts deleted file mode 100644 index 0f3bb0b61f..0000000000 --- a/src/firmware/FirmwareProvider.test.ts +++ /dev/null @@ -1,98 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; - -import { projectAnalysisError } from "../domain/analysisErrorProjection.js"; -import { ProviderAdapterError } from "../domain/providerAdapterError.js"; -import { ProviderCleanupError } from "../domain/providerCleanupError.js"; -import { err, ok } from "../domain/result.js"; -import { firmwareRequestSchema } from "../domain/firmware/firmwareAnalysis.js"; -import { finishFirmwareWorkspace } from "./FirmwareProvider.js"; - -const request = firmwareRequestSchema.parse({ - operation: "extract_firmware", - input: { - path: "/tmp/firmware.bin", - output_directory: "/tmp/extracted", - }, -}); - -const execution = ok({ - result: { files: ["kept"] }, - rawResult: null, - provider: { id: "unblob", name: "unblob", version: "1" }, - limitations: [], - locations: [], - subject: null, -}); - -describe("firmware workspace cleanup", () => { - it("returns the completed extraction beside a cleanup failure and does not delete the leaked root", async () => { - const close = vi.fn(async () => { - throw new Error("cleanup failed"); - }); - const finished = await finishFirmwareWorkspace( - { path: "/tmp/runtime", close }, - execution, - "unblob", - request, - ); - expect(close).toHaveBeenCalledOnce(); - if (finished.ok || !(finished.error instanceof ProviderAdapterError)) - throw new Error("Expected cleanup failure"); - expect(finished.error.cleanupResources).toEqual([ - "/tmp/runtime", - "/tmp/extracted", - ]); - expect(finished.error.partialObservation).toEqual({ - kind: "firmware", - result: { files: ["kept"] }, - }); - expect(finished.error.diagnostics).toEqual({ reason: "cleanup failed" }); - expect(projectAnalysisError(finished.error).details).toMatchObject({ - partial_observation: { - kind: "firmware", - result: { files: ["kept"] }, - }, - }); - }); - - it("does not replay a cleanup failure for a later workspace", async () => { - const failed = await finishFirmwareWorkspace( - { - path: "/tmp/runtime", - close: async () => { - throw new Error("cleanup failed"); - }, - }, - execution, - "unblob", - request, - ); - expect(failed.ok).toBe(false); - const next = await finishFirmwareWorkspace( - { path: "/tmp/next", close: async () => undefined }, - execution, - "unblob", - request, - ); - expect(next).toBe(execution); - }); - - it("leaves an uncertain workspace recorded and does not remove it", async () => { - const close = vi.fn(async () => undefined); - const cleanup = new ProviderCleanupError( - "unblob", - ["/tmp/runtime"], - { reason: "ownership uncertain" }, - { operation: "extract_firmware" }, - ); - const outcome = err(cleanup); - const finished = await finishFirmwareWorkspace( - { path: "/tmp/runtime", close }, - outcome, - "unblob", - request, - ); - expect(close).not.toHaveBeenCalled(); - expect(finished).toBe(outcome); - }); -}); diff --git a/src/firmware/FirmwareProvider.ts b/src/firmware/FirmwareProvider.ts index 8be7901404..6f21f22887 100644 --- a/src/firmware/FirmwareProvider.ts +++ b/src/firmware/FirmwareProvider.ts @@ -3,13 +3,16 @@ import { z } from "zod"; import type { FirmwareAnalysisPort } from "../application/firmware/FirmwareAnalysisPort.js"; import { createAnalysisExecution, + type AnalysisExecution, type ExecutionOptions, } from "../application/AnalysisProvider.js"; import { firmwareResultSchemas, type FirmwareRequest, + type FirmwarePartialObservation, } from "../domain/firmware/firmwareAnalysis.js"; import { AnalysisError } from "../domain/analysisErrorBase.js"; +import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js"; import { AnalysisCancelledError, AnalysisInputError, @@ -18,6 +21,7 @@ import { } from "../domain/analysisErrorCore.js"; import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { ProviderCleanupError } from "../domain/providerCleanupError.js"; +import { jsonValueSchema } from "../domain/jsonValue.js"; import { err, ok } from "../domain/result.js"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import type { ProviderProcessSupervisor } from "../process/ProviderProcess.js"; @@ -42,6 +46,37 @@ import { } from "./FirmwareReports.js"; type Outcome = Awaited>; +interface PendingFirmwareCleanup { + supervisor?: ProviderProcessSupervisor; + readonly root: PrivateRuntimeRoot; + readonly engine: string; + readonly operation: string; +} + +interface FirmwareCleanupAttempt { + readonly failure: ProviderCleanupError | undefined; + readonly processUnverified: boolean; +} + +const combineCleanupFailures = ( + failures: readonly ProviderCleanupError[], +): ProviderCleanupError | undefined => { + if (failures.length < 2) return failures[0]; + return new ProviderCleanupError( + "firmware", + failures.flatMap((failure) => [...failure.cleanupResources]), + { + reason: "Multiple owned firmware resources could not be cleaned", + failures: failures.map((failure) => ({ + engine: failure.providerId, + operation: failure.operation, + resources: [...failure.cleanupResources], + reason: failure.diagnostics?.reason ?? failure.message, + })), + }, + { operation: "close_firmware", cause: new AggregateError(failures) }, + ); +}; const waitForPredecessor = ( predecessor: Promise, operation: string, @@ -67,14 +102,7 @@ const limitations = [ export class FirmwareProvider implements FirmwareAnalysisPort { #tail: Promise = Promise.resolve(); #closed = false; - #pendingCleanup: - | { - supervisor?: ProviderProcessSupervisor; - root: PrivateRuntimeRoot; - engine: string; - operation: string; - } - | undefined; + readonly #pendingCleanup = new Map(); constructor( readonly environment: Readonly< Record @@ -107,53 +135,71 @@ export class FirmwareProvider implements FirmwareAnalysisPort { try { if (this.#closed) return err(new AnalysisCancelledError(request.operation)); - try { - await this.#retryCleanup(); - } catch (cause: unknown) { - return err( - cause instanceof AnalysisError - ? cause - : new ProviderCleanupError("firmware", [], { - reason: cause instanceof Error ? cause.message : String(cause), - }), - ); - } + const cleanup = await this.#retryCleanup(); + if (cleanup.processUnverified && cleanup.failure !== undefined) + return err(cleanup.failure); return await this.#execute(request, options); } finally { release(); } } - /** Retry identity-bound cleanup before relinquishing private workspaces. */ + /** Retry every retained owner, reporting failures without discarding other roots. */ async close(): Promise { this.#closed = true; await this.#tail; - await this.#retryCleanup(); + const cleanup = await this.#retryCleanup(); + if (cleanup.failure !== undefined) throw cleanup.failure; } - async #retryCleanup(): Promise { - const pending = this.#pendingCleanup; - if (pending === undefined) return; - const stopped = await pending.supervisor?.stop(); - if (stopped?.status === "incomplete") - throw new ProviderCleanupError( - pending.engine, - [pending.root.path], - { reason: stopped.reason }, - { operation: pending.operation }, - ); - delete pending.supervisor; - try { - await pending.root.close(); - } catch (cause: unknown) { - throw new ProviderCleanupError( - pending.engine, - [pending.root.path], - { reason: cause instanceof Error ? cause.message : String(cause) }, - { operation: pending.operation, cause }, - ); + async #retryCleanup(): Promise { + const failures: ProviderCleanupError[] = []; + let processUnverified = false; + for (const [path, pending] of this.#pendingCleanup) { + try { + const stopped = await pending.supervisor?.stop(); + if (stopped?.status === "incomplete") { + processUnverified = true; + failures.push( + new ProviderCleanupError( + pending.engine, + [path], + { reason: stopped.reason }, + { operation: pending.operation }, + ), + ); + continue; + } + delete pending.supervisor; + } catch (cause: unknown) { + processUnverified = true; + failures.push( + new ProviderCleanupError( + pending.engine, + [path], + { reason: cause instanceof Error ? cause.message : String(cause) }, + { operation: pending.operation, cause }, + ), + ); + continue; + } + try { + await pending.root.close(); + this.#pendingCleanup.delete(path); + } catch (cause: unknown) { + // A verified stopped process cannot use this root. Keep ownership for + // later removal without blocking a new operation's disjoint workspace. + failures.push( + new ProviderCleanupError( + pending.engine, + [path], + { reason: cause instanceof Error ? cause.message : String(cause) }, + { operation: pending.operation, cause }, + ), + ); + } } - this.#pendingCleanup = undefined; + return { failure: combineCleanupFailures(failures), processUnverified }; } async #execute( @@ -192,12 +238,12 @@ export class FirmwareProvider implements FirmwareAnalysisPort { environment: this.environment, signal, retainCleanup: (supervisor) => { - this.#pendingCleanup = { + this.#pendingCleanup.set(ownedRoot.path, { supervisor, root: ownedRoot, engine: engineName, operation: request.operation, - }; + }); }, ...(this.launcher === undefined ? {} : { launcher: this.launcher }), ...(outputBudget === undefined @@ -386,12 +432,17 @@ export class FirmwareProvider implements FirmwareAnalysisPort { engineName, request, ); - if (!finished.ok && finished.error.cleanupIncomplete && root !== undefined) - this.#pendingCleanup ??= { + if ( + !finished.ok && + finished.error.cleanupIncomplete && + root !== undefined && + !this.#pendingCleanup.has(root.path) + ) + this.#pendingCleanup.set(root.path, { root, engine: engineName, operation: request.operation, - }; + }); return finished; } } @@ -429,32 +480,53 @@ export const finishFirmwareWorkspace = async ( try { await root?.close(); } catch (cause: unknown) { + const cleanup = new ProviderCleanupError( + engineName, + [ + root?.path ?? "unknown", + ...(outcome.ok && request.operation === "extract_firmware" + ? [request.input.output_directory] + : []), + ], + { reason: cause instanceof Error ? cause.message : String(cause) }, + { + cause, + operation: request.operation, + ...(outcome.ok + ? { + partialObservation: completedFirmwareObservation(outcome.value), + } + : outcome.error.partialObservation === undefined + ? {} + : { partialObservation: outcome.error.partialObservation }), + }, + ); return err( - new ProviderCleanupError( - engineName, - [ - root?.path ?? "unknown", - ...(outcome.ok && request.operation === "extract_firmware" - ? [request.input.output_directory] - : []), - ], - { reason: cause instanceof Error ? cause.message : String(cause) }, - { - cause, - operation: request.operation, - ...(outcome.ok - ? { - partialObservation: { - kind: "firmware", - result: outcome.value.result, - }, - } - : outcome.error.partialObservation === undefined - ? {} - : { partialObservation: outcome.error.partialObservation }), - }, - ), + outcome.ok + ? cleanup + : analysisErrorWithCleanupFailure( + outcome.error, + cleanup, + request.operation, + ), ); } return outcome; }; + +const completedFirmwareObservation = ( + execution: AnalysisExecution, +): FirmwarePartialObservation => ({ + kind: "firmware", + result: execution.result, + provenance: jsonValueSchema.parse({ + raw_result: execution.rawResult, + provider: execution.provider, + subject: execution.subject, + locations: execution.locations, + limitations: execution.limitations, + ...(execution.analysisProfile === undefined + ? {} + : { analysis_profile: execution.analysisProfile }), + }), +}); diff --git a/src/generatedPackageMetadata.ts b/src/generatedPackageMetadata.ts index aaa5b9a805..ab4334d0a3 100644 --- a/src/generatedPackageMetadata.ts +++ b/src/generatedPackageMetadata.ts @@ -5,5 +5,5 @@ export const PACKAGE_METADATA = { serverSdkVersion: "2.3.1", clientSdkVersion: "2.3.1", coreSdkVersion: "2.3.1", - skillVersion: "36", + skillVersion: "37", } as const; diff --git a/src/ghidra/GhidraClient.ts b/src/ghidra/GhidraClient.ts index 33c5add1b9..1c1f262bda 100644 --- a/src/ghidra/GhidraClient.ts +++ b/src/ghidra/GhidraClient.ts @@ -24,7 +24,10 @@ import type { } from "./GhidraClientTypes.js"; import type { GhidraInventoryOperation } from "./GhidraInventoryValues.js"; import type { GhidraFunctionOperation } from "./GhidraFunctionValues.js"; -import { createGhidraDiagnostics } from "./GhidraDiagnostics.js"; +import { + createGhidraDiagnostics, + GHIDRA_PROCESS_DIAGNOSTIC_BYTES, +} from "./GhidraDiagnostics.js"; import { GhidraLaunchError, type GhidraLaunch } from "./GhidraLauncher.js"; import { GhidraResponseBuffer } from "./GhidraResponseBuffer.js"; import { GhidraResponseRouter } from "./GhidraResponseRouter.js"; @@ -432,6 +435,7 @@ export class GhidraClient { if (partialLaunch !== undefined) { this.#launch = partialLaunch; this.#process = new ProviderProcessSupervisor(partialLaunch, { + maxDiagnosticBytes: GHIDRA_PROCESS_DIAGNOSTIC_BYTES, onDiagnostic: (event) => this.#onProcessDiagnostic(event), }); } @@ -444,6 +448,7 @@ export class GhidraClient { } this.#launch = launched.value; this.#process = new ProviderProcessSupervisor(launched.value, { + maxDiagnosticBytes: GHIDRA_PROCESS_DIAGNOSTIC_BYTES, onDiagnostic: (event) => this.#onProcessDiagnostic(event), }); const connected = await this.#connect(endpoint, deadline); diff --git a/src/ghidra/GhidraDiagnostics.ts b/src/ghidra/GhidraDiagnostics.ts index 444dac5d23..2530751fcd 100644 --- a/src/ghidra/GhidraDiagnostics.ts +++ b/src/ghidra/GhidraDiagnostics.ts @@ -1,8 +1,12 @@ import type { JsonValue } from "../domain/jsonValue.js"; +import { redactCapturedTransportCredential } from "../process/ProviderDiagnosticRedaction.js"; import type { ProviderProcessSnapshot } from "../process/ProviderProcess.js"; import type { GhidraLaunch } from "./GhidraLauncher.js"; import type { GhidraTransportKind } from "./GhidraTransport.js"; +/** Aggregate in-memory retention budget for this provider's launcher output. */ +export const GHIDRA_PROCESS_DIAGNOSTIC_BYTES = 8 * 1024 * 1024; + /** Inputs needed to project bounded diagnostics for one Ghidra runtime. */ export interface GhidraDiagnosticsOptions { readonly targetPath: string; @@ -31,10 +35,6 @@ export const createGhidraDiagnostics = ( Object.keys(options.previous).length > 0 ) return options.previous; - const redacted = (value: string): string => - options.token === undefined - ? value - : value.replaceAll(options.token, "[REDACTED]"); return { target_path: options.targetPath, target_sha256: options.targetSha256, @@ -67,16 +67,19 @@ export const createGhidraDiagnostics = ( : { exit_code: options.snapshot.exitCode ?? null, exit_signal: options.snapshot.signal ?? null, - stdout: boundedStream(options.snapshot.stdout, redacted), - stderr: boundedStream(options.snapshot.stderr, redacted), + diagnostic_truncated: options.snapshot.diagnosticTruncated === true, + stdout: boundedStream(options.snapshot.stdout, options.token), + stderr: boundedStream(options.snapshot.stderr, options.token), }), }; }; const boundedStream = ( stream: ProviderProcessSnapshot["stdout"], - redact: (value: string) => string, + token: string | undefined, ): JsonValue => ({ - text: redact(stream.text), - bytes: stream.bytes, + text: redactCapturedTransportCredential(stream, token, "[REDACTED]"), + // Preserve the historical total-byte counter; text is now bounded separately. + bytes: stream.observedBytes, + retained_bytes: stream.bytes, }); diff --git a/src/hopper/HopperClient.ts b/src/hopper/HopperClient.ts index 14c5d6965e..c001c3e5ee 100644 --- a/src/hopper/HopperClient.ts +++ b/src/hopper/HopperClient.ts @@ -35,6 +35,7 @@ import { silentLogger } from "../logger.js"; import type { Logger } from "pino"; import { PrivateRuntimeRoot } from "../process/PrivateRuntimeRoot.js"; import { ProviderStartupDeadline } from "../process/ProviderDeadline.js"; +import { redactCapturedTransportCredential } from "../process/ProviderDiagnosticRedaction.js"; import { ProviderRunLineage } from "../process/ProviderRunLineage.js"; import { type ProviderProcessDiagnostic, @@ -64,6 +65,9 @@ import { responseResult, } from "./protocol.js"; +/** Aggregate in-memory retention budget for this provider's launcher output. */ +export const HOPPER_PROCESS_DIAGNOSTIC_BYTES = 8 * 1024 * 1024; + const SHUTDOWN_TIMEOUT_MS = 30_000; const SESSION_ROOT = process.platform === "darwin" ? "/tmp" : tmpdir(); @@ -609,15 +613,27 @@ export class HopperClient { const snapshot = this.#resources.processSupervisor?.snapshot(); const token = this.#token; if (snapshot === undefined) return undefined; - const redact = (text: string): string => - token === undefined - ? text - : text.replaceAll(token, "[redacted transport credential]"); return { exit_code: snapshot.exitCode ?? null, signal: snapshot.signal ?? null, - stdout: { ...snapshot.stdout, text: redact(snapshot.stdout.text) }, - stderr: { ...snapshot.stderr, text: redact(snapshot.stderr.text) }, + stdout: { + text: redactCapturedTransportCredential( + snapshot.stdout, + token, + "[redacted transport credential]", + ), + bytes: snapshot.stdout.observedBytes, + retained_bytes: snapshot.stdout.bytes, + }, + stderr: { + text: redactCapturedTransportCredential( + snapshot.stderr, + token, + "[redacted transport credential]", + ), + bytes: snapshot.stderr.observedBytes, + retained_bytes: snapshot.stderr.bytes, + }, output_closed: outputClosed, diagnostic_truncated: snapshot.diagnosticTruncated === true, }; @@ -743,6 +759,7 @@ export class HopperClient { #attachLauncher(launch: BridgeLaunch): void { this.#resources.processSupervisor = new ProviderProcessSupervisor(launch, { + maxDiagnosticBytes: HOPPER_PROCESS_DIAGNOSTIC_BYTES, onDiagnostic: (event) => this.#onLauncherDiagnostic(launch, event), }); } diff --git a/src/process/DarwinProcessRunTokenReader.ts b/src/process/DarwinProcessRunTokenReader.ts index 85b889bba9..cc837b0281 100644 --- a/src/process/DarwinProcessRunTokenReader.ts +++ b/src/process/DarwinProcessRunTokenReader.ts @@ -262,11 +262,17 @@ export const createDarwinProcessRunTokenReader = ( const read = async ( processes: readonly ProcessTableEntry[], + signal?: AbortSignal, ): Promise> => { + signal?.throwIfAborted(); if (processes.length === 0) return new Map(); - const binary = await compile(); + const binary = await compile(signal); const pids = processes.map(({ pid }) => String(pid)); - const { stdout } = await execFileOutput(binary, pids, { timeout: 15_000 }); + const { stdout } = await execFileOutput(binary, pids, { + timeout: 15_000, + ...(signal === undefined ? {} : { signal }), + }); + signal?.throwIfAborted(); const decoded = safeParseJson(stdout); if (!decoded.ok) throw new Error( diff --git a/src/process/ProcessOwnership.identity.test.ts b/src/process/ProcessOwnership.identity.test.ts index 3d79b281d4..8f6ea72239 100644 --- a/src/process/ProcessOwnership.identity.test.ts +++ b/src/process/ProcessOwnership.identity.test.ts @@ -377,7 +377,7 @@ describe("process identity during token validation", () => { expect(reusedDuringFinalRead).toBe(true); }); - it("fails closed when a candidate PID changes identity during token reading", async () => { + it("fails closed when candidate PIDs change identity during token reading", async () => { let identityReads = 0; const process: ProcessTableEntry = { pid: 911, @@ -386,8 +386,10 @@ describe("process identity during token validation", () => { state: "S", command: "reused-during-token-read", }; + const processes = [process, { ...process, pid: 912, processGroupId: 912 }]; + const listProcesses = vi.fn(async () => processes); const host: ProcessOwnershipHost = { - listProcesses: () => Promise.resolve([process]), + listProcesses, environment: () => Promise.resolve({}), processIdentities: (entries) => { identityReads += 1; @@ -414,21 +416,24 @@ describe("process identity during token validation", () => { }; await expect( - verifyNoTokenOwnedProcesses("run-token", host, [ - { pid: process.pid, identity: "old-start" }, - ]), + verifyNoTokenOwnedProcesses( + "run-token", + host, + processes.map(({ pid }) => ({ pid, identity: "old-start" })), + ), ).resolves.toEqual({ cleaned: false, reason: "process identity could not be revalidated", - failures: [ - { - pid: process.pid, - reason: "process-identity-unavailable", - diagnostic: - "process identity changed or became unavailable during token validation", - }, - ], + failures: processes.map(({ pid }) => ({ + pid, + reason: "process-identity-unavailable", + diagnostic: + "process identity changed or became unavailable during token validation", + })), }); + // Initial observation, final token classification, then one fresh recheck + // shared by both changed identities. + expect(listProcesses).toHaveBeenCalledTimes(3); }); }); diff --git a/src/process/ProcessOwnership.observation.test.ts b/src/process/ProcessOwnership.observation.test.ts new file mode 100644 index 0000000000..58085eb21a --- /dev/null +++ b/src/process/ProcessOwnership.observation.test.ts @@ -0,0 +1,328 @@ +import { describe, expect, it, vi } from "vitest"; +import { + cleanupOwnedProcessGroup, + type ProcessOwnershipHost, +} from "./ProcessOwnership.js"; +import { + observeOwnedProcessGroups, + observeOwnedProcessLineage, +} from "./ProcessOwnershipObservation.js"; +import { host, ownership } from "./ProcessOwnership.fixture.js"; + +const processes = [100, 101].map((pid) => ({ + pid, + parentPid: pid === 100 ? 1 : 100, + processGroupId: 100, + state: "S", + command: "fixture", +})); + +describe("multi-group ownership observation", () => { + it("observes multiple groups from one process snapshot and token batch", async () => { + const rows = [ + ...processes, + { + pid: 200, + parentPid: 1, + processGroupId: 200, + state: "S", + command: "other-owned-group", + }, + { + pid: 300, + parentPid: 1, + processGroupId: 300, + state: "Z", + command: "[exited]", + }, + ]; + const listProcesses = vi.fn(() => Promise.resolve(rows)); + const environment = vi.fn(() => Promise.resolve({})); + const runTokens = vi.fn>( + (members) => + Promise.resolve( + new Map( + members.map(({ pid }) => [ + pid, + { + state: "readable" as const, + runId: pid === 200 ? "other-run" : ownership.runId, + }, + ]), + ), + ), + ); + + await expect( + observeOwnedProcessGroups(ownership.runId, [100, 200, 300], { + listProcesses, + environment, + runTokens, + signalGroup: vi.fn(), + }), + ).resolves.toEqual( + new Map([ + [100, { state: "alive" }], + [ + 200, + { state: "unverifiable", reason: "process ownership did not match" }, + ], + [300, { state: "empty" }], + ]), + ); + expect(listProcesses).toHaveBeenCalledTimes(1); + expect(runTokens).toHaveBeenCalledTimes(1); + expect(runTokens).toHaveBeenCalledWith( + rows.filter(({ state }) => state !== "Z"), + ); + expect(environment).not.toHaveBeenCalled(); + }); + + it("rechecks unavailable members individually before classifying their groups", async () => { + const rows = [ + ...processes, + { + pid: 200, + parentPid: 1, + processGroupId: 200, + state: "S", + command: "unreadable-group", + }, + ]; + const listProcesses = vi + .fn() + .mockResolvedValueOnce(rows) + .mockResolvedValueOnce(rows.filter(({ pid }) => pid !== 101)) + .mockResolvedValueOnce(rows); + const runTokens = vi.fn>( + (members) => + Promise.resolve( + new Map( + members.map(({ pid }) => [ + pid, + pid === 101 || pid === 200 + ? { + state: "unavailable" as const, + reason: `unreadable-${String(pid)}`, + } + : { state: "readable" as const, runId: ownership.runId }, + ]), + ), + ), + ); + + await expect( + observeOwnedProcessGroups(ownership.runId, [100, 200], { + listProcesses, + environment: vi.fn(() => Promise.resolve({})), + runTokens, + signalGroup: vi.fn(), + }), + ).resolves.toEqual( + new Map([ + [100, { state: "alive" }], + [ + 200, + { + state: "unverifiable", + reason: + "process ownership could not be revalidated for PID 200: unreadable-200", + }, + ], + ]), + ); + expect(listProcesses).toHaveBeenCalledTimes(3); + expect(runTokens).toHaveBeenCalledTimes(1); + }); +}); + +describe("multi-group observation short-circuit", () => { + it("stops missing-row fallbacks once every live group is unverifiable", async () => { + const listProcesses = vi.fn(() => Promise.resolve(processes)); + const environment = vi.fn((pid: number) => + Promise.resolve({ + REA_PROCESS_RUN_ID: pid === 100 ? "other-run" : ownership.runId, + }), + ); + const runTokens = vi.fn>( + () => Promise.resolve(new Map()), + ); + + await expect( + observeOwnedProcessGroups(ownership.runId, [100], { + listProcesses, + environment, + runTokens, + signalGroup: vi.fn(), + }), + ).resolves.toEqual( + new Map([ + [ + 100, + { state: "unverifiable", reason: "process ownership did not match" }, + ], + ]), + ); + expect(runTokens).toHaveBeenCalledTimes(1); + expect(environment.mock.calls).toEqual([[100]]); + expect(listProcesses).toHaveBeenCalledTimes(1); + }); +}); + +describe("batched ownership observation", () => { + it("observes group and lineage with one batch each and falls back only for missing rows", async () => { + const { adapter } = host({ + 100: { REA_PROCESS_RUN_ID: ownership.runId }, + 101: { REA_PROCESS_RUN_ID: ownership.runId }, + }); + const environment = vi.fn(adapter.environment); + const runTokens = vi.fn>( + () => + Promise.resolve( + new Map([[100, { state: "readable", runId: ownership.runId }]]), + ), + ); + const observedHost = { ...adapter, environment, runTokens }; + await expect( + observeOwnedProcessGroups( + ownership.runId, + [ownership.processGroupId], + observedHost, + ), + ).resolves.toEqual( + new Map([[ownership.processGroupId, { state: "alive" }]]), + ); + await expect( + observeOwnedProcessLineage(ownership, observedHost), + ).resolves.toMatchObject({ + status: "verified", + lineage: { + descendants: [{ pid: 101, parentPid: 100, processGroupId: 100 }], + }, + }); + expect(runTokens.mock.calls).toEqual([[processes], [processes]]); + expect(environment.mock.calls).toEqual([[101], [101]]); + }); + + it.each(["batch", "environment"] as const)( + "preserves cancellation during %s reads, including successful completion races", + async (boundary) => { + const controller = new AbortController(); + const reason = new Error("cancelled ownership read"); + const listProcesses = vi.fn(() => Promise.resolve(processes)); + const environment = vi.fn( + async (_pid, signal) => { + expect(signal).toBe(controller.signal); + controller.abort(reason); + return { REA_PROCESS_RUN_ID: ownership.runId }; + }, + ); + const runTokens = vi.fn>( + async (_processes, signal) => { + expect(signal).toBe(controller.signal); + if (boundary === "batch") controller.abort(reason); + return new Map(); + }, + ); + await expect( + observeOwnedProcessGroups( + ownership.runId, + [ownership.processGroupId], + { listProcesses, environment, runTokens, signalGroup: vi.fn() }, + controller.signal, + ), + ).rejects.toBe(reason); + expect(listProcesses).toHaveBeenCalledTimes(1); + expect(environment).toHaveBeenCalledTimes(boundary === "batch" ? 0 : 1); + }, + ); + + it.each([100, 101])( + "rechecks unavailable batch rows for exited PID %i without overriding them through fallback", + async (exitedPid) => { + const listProcesses = vi + .fn() + .mockResolvedValueOnce(processes) + .mockResolvedValue(processes.filter(({ pid }) => pid !== exitedPid)); + const environment = vi.fn(() => + Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }), + ); + const observedHost: ProcessOwnershipHost = { + listProcesses, + environment, + signalGroup: vi.fn(), + runTokens: () => + Promise.resolve( + new Map( + processes.map(({ pid }) => [ + pid, + pid === exitedPid + ? { state: "unavailable" as const, reason: "process exited" } + : { state: "readable" as const, runId: ownership.runId }, + ]), + ), + ), + }; + const result = await observeOwnedProcessLineage(ownership, observedHost); + if (exitedPid === 100) + expect(result).toMatchObject({ + status: "unavailable", + reason: "owned launcher exited during lineage validation", + }); + else + expect(result).toMatchObject({ + status: "verified", + lineage: { descendants: [] }, + }); + expect(environment).not.toHaveBeenCalled(); + }, + ); +}); + +describe("token sweep batch failures", () => { + it("cleans token-owned groups when a failed batch has readable fallback observations", async () => { + const { adapter, signalGroup } = host({ + 100: { REA_PROCESS_RUN_ID: ownership.runId }, + 101: { REA_PROCESS_RUN_ID: ownership.runId }, + }); + await expect( + cleanupOwnedProcessGroup( + { ...ownership, sweepTokenOwnedProcesses: true, captureBaseline: [] }, + { + ...adapter, + runTokens: () => + Promise.reject(new Error("batch reader unavailable")), + }, + ), + ).resolves.toEqual({ cleaned: true, signaled: true }); + expect(signalGroup.mock.calls).toEqual([[100, "SIGKILL"]]); + }); + + it("uses fallback ownership when the initial batch fails and preserves both failures through retries", async () => { + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: () => Promise.resolve(processes), + environment: (pid) => + pid === 100 + ? Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }) + : Promise.reject(new Error("individual unreadable")), + runTokens: () => Promise.reject(new Error("batch reader unavailable")), + signalGroup, + }; + const result = await cleanupOwnedProcessGroup( + { ...ownership, sweepTokenOwnedProcesses: true, captureBaseline: [] }, + adapter, + ); + expect(result).toMatchObject({ + cleaned: false, + failures: expect.arrayContaining([ + expect.objectContaining({ + pid: 101, + diagnostic: + "individual unreadable; run-token batch failed: batch reader unavailable", + }), + ]), + }); + expect(signalGroup).not.toHaveBeenCalled(); + }); +}); diff --git a/src/process/ProcessOwnership.ts b/src/process/ProcessOwnership.ts index e0330b9162..db53e08c72 100644 --- a/src/process/ProcessOwnership.ts +++ b/src/process/ProcessOwnership.ts @@ -7,6 +7,7 @@ import { } from "./ProcessOwnershipObservation.js"; import { launcherIdentityFailure } from "./ProcessOwnershipIdentity.js"; import { descendantsOf, liveProcesses } from "./ProcessOwnershipProcessTree.js"; +import { readProcessRunTokens } from "./ProcessRunTokenObservations.js"; const execFileAsync = promisify(execFile); @@ -90,10 +91,14 @@ export interface ProcessOwnershipHost { close?(): Promise; /** List current processes; startup inspection may be cancelled by the caller. */ listProcesses(signal?: AbortSignal): Promise; - environment(pid: number): Promise>>; + environment( + pid: number, + signal?: AbortSignal, + ): Promise>>; /** Read only run-token values in one host operation when the OS supports it. */ runTokens?( processes: readonly ProcessTableEntry[], + signal?: AbortSignal, ): Promise>; /** Read stable identities; omit the signal during cleanup so it can finish. */ processIdentities?( @@ -623,6 +628,7 @@ const processOwnershipFailures = async ( expectedIdentities.has(pid), ); const observed = await host.processIdentities(expectedMembers); + const identityFailures: ProcessOwnershipValidationFailure[] = []; for (const member of expectedMembers) { const expected = expectedIdentities.get(member.pid); const current = observed.get(member.pid); @@ -631,8 +637,7 @@ const processOwnershipFailures = async ( current?.state !== "readable" || current.identity !== expected ) { - if (await processIsGone(host, member.pid)) continue; - failures.push({ + identityFailures.push({ pid: member.pid, reason: "process-identity-unavailable", diagnostic: @@ -640,30 +645,9 @@ const processOwnershipFailures = async ( }); } } + failures.push(...(await recheckOwnershipFailures(identityFailures, host))); } - for (const member of members) { - try { - if ((await host.environment(member.pid)).REA_PROCESS_RUN_ID !== runId) - failures.push({ pid: member.pid, reason: "run-token-mismatch" }); - } catch (cause: unknown) { - try { - const live = liveProcesses(await host.listProcesses()); - if (!live.some(({ pid }) => pid === member.pid)) continue; - } catch (recheckCause: unknown) { - failures.push({ - pid: member.pid, - reason: "environment-unreadable", - diagnostic: `${errorMessage(cause)}; process liveness recheck failed: ${errorMessage(recheckCause)}`, - }); - continue; - } - failures.push({ - pid: member.pid, - reason: "environment-unreadable", - diagnostic: errorMessage(cause), - }); - } - } + failures.push(...(await processRunTokenFailures(members, runId, host))); if ( failures.length === 0 && expectedIdentities.size > 0 && @@ -673,6 +657,7 @@ const processOwnershipFailures = async ( expectedIdentities.has(pid), ); const observed = await host.processIdentities(expectedMembers); + const identityFailures: ProcessOwnershipValidationFailure[] = []; for (const member of expectedMembers) { const expected = expectedIdentities.get(member.pid); const current = observed.get(member.pid); @@ -682,18 +667,72 @@ const processOwnershipFailures = async ( current.identity === expected ) continue; - if (await processIsGone(host, member.pid)) continue; - failures.push({ + identityFailures.push({ pid: member.pid, reason: "process-identity-unavailable", diagnostic: "process identity changed or became unavailable during token validation", }); } + failures.push(...(await recheckOwnershipFailures(identityFailures, host))); } return failures; }; +const processRunTokenFailures = async ( + members: readonly ProcessTableEntry[], + runId: string, + host: ProcessOwnershipHost, +): Promise => { + const failures: ProcessOwnershipValidationFailure[] = []; + for await (const { process: member, observation } of readProcessRunTokens( + host, + members, + )) { + if (observation.state === "readable") { + if (observation.runId !== runId) + failures.push({ pid: member.pid, reason: "run-token-mismatch" }); + continue; + } + failures.push({ + pid: member.pid, + reason: "environment-unreadable", + diagnostic: observation.reason, + }); + } + return recheckOwnershipFailures(failures, host); +}; + +/** + * Take a fresh liveness snapshot after each ownership-read phase. Only an + * absent PID resolves an unreadable observation; a readable token mismatch + * remains a failure. Never reuse this snapshot across identity/token reads. + */ +const recheckOwnershipFailures = async ( + failures: readonly ProcessOwnershipValidationFailure[], + host: ProcessOwnershipHost, +): Promise => { + if (!failures.some(({ reason }) => reason !== "run-token-mismatch")) + return failures; + try { + const live = new Set( + liveProcesses(await host.listProcesses()).map(({ pid }) => pid), + ); + return failures.filter( + ({ pid, reason }) => reason === "run-token-mismatch" || live.has(pid), + ); + } catch (cause: unknown) { + return failures.map((failure) => + failure.reason === "run-token-mismatch" + ? failure + : { + ...failure, + diagnostic: `${failure.diagnostic}; process liveness recheck failed: ${errorMessage(cause)}`, + }, + ); + } +}; + interface TokenOwnedProcessScan { readonly owned: readonly ProcessTableEntry[]; readonly failures: readonly ProcessOwnershipValidationFailure[]; @@ -767,18 +806,19 @@ const scanTokenOwnedProcesses = async ( const failures: ProcessOwnershipValidationFailure[] = []; if (captureBaseline !== undefined && host.processIdentities !== undefined) { const stableCandidates: ProcessTableEntry[] = []; + const identityFailures: ProcessOwnershipValidationFailure[] = []; for (const process of candidates) { if (candidateIdentities.get(process.pid)?.state === "readable") { stableCandidates.push(process); continue; } - if (!(await processIsGone(host, process.pid))) - failures.push({ - pid: process.pid, - reason: "process-identity-unavailable", - diagnostic: "process identity was unavailable during token scan", - }); + identityFailures.push({ + pid: process.pid, + reason: "process-identity-unavailable", + diagnostic: "process identity was unavailable during token scan", + }); } + failures.push(...(await recheckOwnershipFailures(identityFailures, host))); candidates = stableCandidates; } const owned: ProcessTableEntry[] = []; @@ -791,26 +831,14 @@ const scanTokenOwnedProcesses = async ( identity?.state === "readable" ? identity.identity : null, ); }; - let bulkTokens: ReadonlyMap | undefined; - if (host.runTokens !== undefined) - bulkTokens = await host.runTokens(candidates); const unreadable: UnreadableTokenCandidate[] = []; - for (const process of candidates) { - const bulkToken = bulkTokens?.get(process.pid); - if (bulkToken !== undefined) { - if (bulkToken.state === "readable") { - if (bulkToken.runId === runId) recordOwned(process); - continue; - } - unreadable.push({ process, diagnostic: bulkToken.reason }); - continue; - } - try { - if ((await host.environment(process.pid)).REA_PROCESS_RUN_ID === runId) - recordOwned(process); - } catch (cause: unknown) { - unreadable.push({ process, diagnostic: errorMessage(cause) }); - } + for await (const { process, observation } of readProcessRunTokens( + host, + candidates, + )) { + if (observation.state === "readable") { + if (observation.runId === runId) recordOwned(process); + } else unreadable.push({ process, diagnostic: observation.reason }); } const remaining = await settleUnreadableTokenCandidates( unreadable, @@ -876,6 +904,7 @@ const scanTokenOwnedProcesses = async ( if (candidates.length > 0 && host.processIdentities !== undefined) { const afterRead = await host.processIdentities(candidates); const stillOwned: ProcessTableEntry[] = []; + const identityFailures: ProcessOwnershipValidationFailure[] = []; const ownedPids = new Set(owned.map(({ pid }) => pid)); for (const process of candidates) { const before = candidateIdentities.get(process.pid); @@ -888,14 +917,14 @@ const scanTokenOwnedProcesses = async ( if (ownedPids.has(process.pid)) stillOwned.push(process); continue; } - if (await processIsGone(host, process.pid)) continue; - failures.push({ + identityFailures.push({ pid: process.pid, reason: "process-identity-unavailable", diagnostic: "process identity changed or became unavailable during token validation", }); } + failures.push(...(await recheckOwnershipFailures(identityFailures, host))); return { owned: stillOwned, failures, @@ -936,31 +965,20 @@ const settleUnreadableTokenCandidates = async ( void cause; live = undefined; } - let observations: - | ReadonlyMap - | undefined; - if (host.runTokens !== undefined) { - try { - observations = await host.runTokens( - remaining.map(({ process }) => process), - ); - } catch (cause: unknown) { - void cause; - observations = undefined; - } - } const stillUnreadable: UnreadableTokenCandidate[] = []; - for (const candidate of remaining) { - const pid = candidate.process.pid; - if (live !== undefined && !live.has(pid)) continue; - const observation = - observations?.get(pid) ?? (await readEnvironmentRunToken(host, pid)); + const candidates = remaining + .map(({ process }) => process) + .filter(({ pid }) => live === undefined || live.has(pid)); + for await (const { process, observation } of readProcessRunTokens( + host, + candidates, + )) { if (observation.state === "readable") { - if (observation.runId === runId) recordOwned(candidate.process); + if (observation.runId === runId) recordOwned(process); continue; } stillUnreadable.push({ - process: candidate.process, + process, diagnostic: observation.reason, }); } @@ -968,31 +986,3 @@ const settleUnreadableTokenCandidates = async ( } return remaining; }; - -const readEnvironmentRunToken = async ( - host: ProcessOwnershipHost, - pid: number, -): Promise => { - try { - return { - state: "readable", - runId: (await host.environment(pid)).REA_PROCESS_RUN_ID, - }; - } catch (cause: unknown) { - return { state: "unavailable", reason: errorMessage(cause) }; - } -}; - -const processIsGone = async ( - host: ProcessOwnershipHost, - pid: number, -): Promise => { - try { - return !liveProcesses(await host.listProcesses()).some( - (process) => process.pid === pid, - ); - } catch (cause: unknown) { - void cause; - return false; - } -}; diff --git a/src/process/ProcessOwnership.validation.test.ts b/src/process/ProcessOwnership.validation.test.ts index 29abdf8768..5364e4e2f7 100644 --- a/src/process/ProcessOwnership.validation.test.ts +++ b/src/process/ProcessOwnership.validation.test.ts @@ -6,7 +6,7 @@ import { type WindowsProcessTreeHost, } from "./ProcessOwnership.js"; import { - observeOwnedProcessGroup, + observeOwnedProcessGroups, observeOwnedProcessLineage, } from "./ProcessOwnershipObservation.js"; import { host, ownership } from "./ProcessOwnership.fixture.js"; @@ -115,6 +115,101 @@ describe("owned process-group cleanup validation: ownership and lineage", () => }); expect(signalGroup).not.toHaveBeenCalled(); }); +}); + +describe("owned process-group cleanup liveness rechecks", () => { + it("rechecks all unreadable members after the token phase in one fresh snapshot", async () => { + const members = [100, 101, 102, 103].map((pid) => ({ + pid, + parentPid: pid === 100 ? 1 : 100, + processGroupId: 100, + state: "S", + command: "fixture", + })); + const tokenReads: number[] = []; + const listProcesses = vi.fn(async () => { + if (tokenReads.length > 0) + expect(tokenReads).toEqual([100, 101, 102, 103]); + return members; + }); + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses, + environment: async (pid) => { + tokenReads.push(pid); + throw new Error(`unreadable ${pid}`); + }, + signalGroup, + }; + await expect(cleanupOwnedProcessGroup(ownership, adapter)).resolves.toEqual( + { + cleaned: false, + reason: + "process ownership token could not be read for 4 live process(es): other_unavailable=4; live candidates 100=other_unavailable, 101=other_unavailable, 102=other_unavailable, 103=other_unavailable", + failures: members.map(({ pid }) => ({ + pid, + reason: "environment-unreadable", + diagnostic: `unreadable ${pid}`, + })), + }, + ); + expect(listProcesses).toHaveBeenCalledTimes(2); + expect(signalGroup).not.toHaveBeenCalled(); + }); + + it.each([false, true])( + "retains mismatches and fails closed when the liveness recheck fails: %s", + async (recheckFails) => { + const members = [100, 101, 102].map((pid) => ({ + pid, + parentPid: pid === 100 ? 1 : 100, + processGroupId: 100, + state: "S", + command: "fixture", + })); + const listProcesses = vi + .fn() + .mockResolvedValueOnce(members); + if (recheckFails) + listProcesses.mockRejectedValue(new Error("process table denied")); + else listProcesses.mockResolvedValue([]); + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses, + environment: async (pid) => { + if (pid === 101) return { REA_PROCESS_RUN_ID: "other-run" }; + throw new Error(`unreadable ${pid}`); + }, + signalGroup, + }; + await expect( + cleanupOwnedProcessGroup(ownership, adapter), + ).resolves.toEqual({ + cleaned: false, + reason: "process tree contains an unowned or PID-reused process", + failures: recheckFails + ? [ + { + pid: 100, + reason: "environment-unreadable", + diagnostic: + "unreadable 100; process liveness recheck failed: process table denied", + }, + { pid: 101, reason: "run-token-mismatch" }, + { + pid: 102, + reason: "environment-unreadable", + diagnostic: + "unreadable 102; process liveness recheck failed: process table denied", + }, + ] + : [{ pid: 101, reason: "run-token-mismatch" }], + }); + expect(listProcesses).toHaveBeenCalledTimes(2); + expect(signalGroup).not.toHaveBeenCalled(); + }, + ); + it("accepts a member that exits during ownership revalidation", async () => { const liveLauncher = { pid: 100, @@ -160,7 +255,12 @@ describe("owned process-group cleanup validation: exited members", () => { }, }; await expect( - observeOwnedProcessGroup(ownership, adapter, controller.signal), + observeOwnedProcessGroups( + ownership.runId, + [ownership.processGroupId], + adapter, + controller.signal, + ), ).rejects.toBe(reason); }); it("does not inspect a process group after observation was cancelled", async () => { @@ -177,7 +277,12 @@ describe("owned process-group cleanup validation: exited members", () => { }, }; await expect( - observeOwnedProcessGroup(ownership, adapter, controller.signal), + observeOwnedProcessGroups( + ownership.runId, + [ownership.processGroupId], + adapter, + controller.signal, + ), ).rejects.toBe(reason); }); it("ignores exited zombie members during live ownership checks", async () => { @@ -229,9 +334,13 @@ describe("owned process-group cleanup validation: exited members", () => { environment, signalGroup: vi.fn(), }; - expect(await observeOwnedProcessGroup(ownership, adapter)).toEqual({ - state: "empty", - }); + expect( + await observeOwnedProcessGroups( + ownership.runId, + [ownership.processGroupId], + adapter, + ), + ).toEqual(new Map([[ownership.processGroupId, { state: "empty" }]])); expect(environment).not.toHaveBeenCalled(); }); it("is idempotent when the owned group has already exited", async () => { @@ -278,6 +387,125 @@ describe("owned process-group cleanup validation: exited members", () => { }); }); +describe("owned process-group cleanup token batches", () => { + const root = { + pid: 100, + parentPid: 1, + processGroupId: 100, + state: "S", + command: "fixture", + }; + + it("batches validation and falls back only for a missing member row", async () => { + const child = { ...root, pid: 101, parentPid: root.pid }; + const runTokens = vi.fn(() => + Promise.resolve( + new Map([ + [root.pid, { state: "readable" as const, runId: ownership.runId }], + ]), + ), + ); + const environment = vi.fn(() => + Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }), + ); + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: () => Promise.resolve([root, child]), + environment, + runTokens, + signalGroup, + }; + + await expect(cleanupOwnedProcessGroup(ownership, adapter)).resolves.toEqual( + { cleaned: true, signaled: true }, + ); + expect(runTokens).toHaveBeenCalledTimes(2); + expect(runTokens).toHaveBeenNthCalledWith(1, [root, child]); + expect(environment.mock.calls).toEqual([[child.pid], [child.pid]]); + expect(signalGroup).toHaveBeenCalledWith(root.processGroupId, "SIGKILL"); + }); + + it("keeps explicit unreadable rows fail-closed without environment fallback", async () => { + const environment = vi.fn(() => + Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }), + ); + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: () => Promise.resolve([root]), + environment, + runTokens: () => + Promise.resolve( + new Map([ + [ + root.pid, + { state: "unavailable" as const, reason: "fixture unreadable" }, + ], + ]), + ), + signalGroup, + }; + + await expect( + cleanupOwnedProcessGroup(ownership, adapter), + ).resolves.toMatchObject({ + cleaned: false, + failures: [ + { + pid: root.pid, + reason: "environment-unreadable", + diagnostic: "fixture unreadable", + }, + ], + }); + expect(environment).not.toHaveBeenCalled(); + expect(signalGroup).not.toHaveBeenCalled(); + }); + + it("falls back to the environment reader when the batch operation throws", async () => { + const environment = vi.fn(() => + Promise.resolve({ REA_PROCESS_RUN_ID: ownership.runId }), + ); + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: () => Promise.resolve([root]), + environment, + runTokens: () => Promise.reject(new Error("batch reader unavailable")), + signalGroup, + }; + + await expect(cleanupOwnedProcessGroup(ownership, adapter)).resolves.toEqual( + { cleaned: true, signaled: true }, + ); + expect(environment).toHaveBeenCalledTimes(2); + expect(signalGroup).toHaveBeenCalledWith(root.processGroupId, "SIGKILL"); + }); + + it("retains both failures when batch and individual token reads fail", async () => { + const signalGroup = vi.fn(); + const adapter: ProcessOwnershipHost = { + listProcesses: () => Promise.resolve([root]), + environment: () => Promise.reject(new Error("individual unreadable")), + runTokens: () => Promise.reject(new Error("batch reader unavailable")), + signalGroup, + }; + + await expect( + cleanupOwnedProcessGroup(ownership, adapter), + ).resolves.toMatchObject({ + cleaned: false, + failures: [ + { + pid: root.pid, + reason: "environment-unreadable", + diagnostic: + "individual unreadable; run-token batch failed: batch reader unavailable", + }, + ], + }); + expect(signalGroup).not.toHaveBeenCalled(); + }); +}); + describe("Windows P0 process-tree cleanup", () => { it("reports whether taskkill signaled or found an exited tree", async () => { const terminated: WindowsProcessTreeHost = { diff --git a/src/process/ProcessOwnershipObservation.ts b/src/process/ProcessOwnershipObservation.ts index 34edeaea92..db5c7cb3a9 100644 --- a/src/process/ProcessOwnershipObservation.ts +++ b/src/process/ProcessOwnershipObservation.ts @@ -5,6 +5,7 @@ import { launcherIdentityFailure } from "./ProcessOwnershipIdentity.js"; import { descendantsOf, liveProcesses } from "./ProcessOwnershipProcessTree.js"; import { execFileOutput } from "./ExecFileOutput.js"; import { createDarwinProcessRunTokenReader } from "./DarwinProcessRunTokenReader.js"; +import { readProcessRunTokens } from "./ProcessRunTokenObservations.js"; import type { OwnedProcessGroup, ProcessGroupObservation, @@ -15,49 +16,78 @@ import type { ProcessTableEntry, } from "./ProcessOwnership.js"; -/** Observe one group without signaling it, failing closed on identity doubt. */ -export const observeOwnedProcessGroup = async ( - ownership: OwnedProcessGroup, +/** Observe multiple groups from one process snapshot and one token batch. */ +export const observeOwnedProcessGroups = async ( + runId: string, + processGroupIds: readonly number[], host: ProcessOwnershipHost = systemProcessOwnershipHost, signal?: AbortSignal, -): Promise => { +): Promise> => { signal?.throwIfAborted(); - let members: readonly ProcessTableEntry[]; + const groupIds = [...new Set(processGroupIds)]; + if (groupIds.length === 0) return new Map(); + let processes: readonly ProcessTableEntry[]; try { - members = (await host.listProcesses(signal)).filter( - ({ processGroupId }) => processGroupId === ownership.processGroupId, - ); + processes = liveProcesses(await host.listProcesses(signal)); } catch (cause: unknown) { signal?.throwIfAborted(); - return { - state: "unverifiable", + const failure = { + state: "unverifiable" as const, reason: `process group could not be inspected: ${errorMessage(cause)}`, }; + return new Map(groupIds.map((groupId) => [groupId, failure])); } signal?.throwIfAborted(); - const liveMembers = liveProcesses(members); - if (liveMembers.length === 0) return { state: "empty" }; - for (const member of liveMembers) { - signal?.throwIfAborted(); - try { - const runId = (await host.environment(member.pid)).REA_PROCESS_RUN_ID; - signal?.throwIfAborted(); - if (runId !== ownership.runId) - return { + const membersByGroup = new Map( + groupIds.map((groupId) => [groupId, [] as ProcessTableEntry[]]), + ); + for (const process of processes) + membersByGroup.get(process.processGroupId)?.push(process); + const observations = new Map( + groupIds.map((groupId) => [ + groupId, + (membersByGroup.get(groupId)?.length ?? 0) > 0 + ? { state: "alive" } + : { state: "empty" }, + ]), + ); + const liveMembers = groupIds.flatMap( + (groupId) => membersByGroup.get(groupId) ?? [], + ); + const pendingGroups = new Set( + groupIds.filter( + (groupId) => (membersByGroup.get(groupId)?.length ?? 0) > 0, + ), + ); + for await (const { process: member, observation } of readProcessRunTokens( + host, + liveMembers, + signal, + )) { + if (!pendingGroups.has(member.processGroupId)) continue; + if (observation.state === "readable") { + if (observation.runId !== runId) { + observations.set(member.processGroupId, { state: "unverifiable", reason: "process ownership did not match", - }; - } catch (cause: unknown) { + }); + pendingGroups.delete(member.processGroupId); + } + } else { + const gone = await processIsGone(host, member.pid); signal?.throwIfAborted(); - if (!(await processIsGone(host, member.pid))) { - return { + if (!gone) { + observations.set(member.processGroupId, { state: "unverifiable", - reason: `process ownership could not be revalidated for PID ${member.pid}: ${errorMessage(cause)}`, - }; + reason: `process ownership could not be revalidated for PID ${member.pid}: ${observation.reason}`, + }); + pendingGroups.delete(member.processGroupId); } } + if (pendingGroups.size === 0) break; } - return { state: "alive" }; + signal?.throwIfAborted(); + return observations; }; /** @@ -88,18 +118,18 @@ export const observeOwnedProcessLineage = async ( return unavailableLineage(ownership, identityFailure); const descendants = descendantsOf(launcher.pid, processes); const verifiedDescendants: ProcessTableEntry[] = []; - for (const member of [launcher, ...descendants]) { - try { - if ( - (await host.environment(member.pid)).REA_PROCESS_RUN_ID !== - ownership.runId - ) + for await (const { process: member, observation } of readProcessRunTokens( + host, + [launcher, ...descendants], + )) { + if (observation.state === "readable") { + if (observation.runId !== ownership.runId) return unavailableLineage( ownership, "process lineage contains an unowned or PID-reused process", ); if (member.pid !== launcher.pid) verifiedDescendants.push(member); - } catch (cause: unknown) { + } else { if (await processIsGone(host, member.pid)) { if (member.pid === launcher.pid) return unavailableLineage( @@ -110,7 +140,7 @@ export const observeOwnedProcessLineage = async ( } return unavailableLineage( ownership, - `process ownership could not be revalidated for PID ${member.pid}: ${errorMessage(cause)}`, + `process ownership could not be revalidated for PID ${member.pid}: ${observation.reason}`, ); } } @@ -330,18 +360,21 @@ export const createSystemProcessOwnershipHost = ( } }, listProcesses, - async environment(pid) { + async environment(pid, signal) { + signal?.throwIfAborted(); if (platform === "linux") return parseProcessEnvironment( - await readFile(`/proc/${pid}/environ`, "utf8"), + await readFile(`/proc/${pid}/environ`, { encoding: "utf8", signal }), ); if (platform === "darwin") { - const process = (await listProcesses()).find( + const process = (await listProcesses(signal)).find( (entry) => entry.pid === pid, ); if (process === undefined) throw new Error(`process ${String(pid)} is not live`); - const observation = (await darwinTokens?.read([process]))?.get(pid); + const observation = (await darwinTokens?.read([process], signal))?.get( + pid, + ); if (observation === undefined || observation.state === "unavailable") throw new Error( observation?.reason ?? "process run token could not be read", @@ -355,6 +388,7 @@ export const createSystemProcessOwnershipHost = ( ["eww", "-p", String(pid)], { env: hostEnvironment, + ...(signal === undefined ? {} : { signal }), }, ); const observedEnvironment: Record = {}; @@ -366,11 +400,13 @@ export const createSystemProcessOwnershipHost = ( } return observedEnvironment; }, - async runTokens(processes) { + async runTokens(processes, signal) { + signal?.throwIfAborted(); if (platform !== "darwin" || processes.length === 0) return new Map(); try { - return (await darwinTokens?.read(processes)) ?? new Map(); + return (await darwinTokens?.read(processes, signal)) ?? new Map(); } catch (cause: unknown) { + if (isExpectedAbort(cause, signal)) throw cause; void cause; return new Map( processes.map(({ pid }) => [ diff --git a/src/process/ProcessRunTokenObservations.ts b/src/process/ProcessRunTokenObservations.ts new file mode 100644 index 0000000000..c0e3b5cdd1 --- /dev/null +++ b/src/process/ProcessRunTokenObservations.ts @@ -0,0 +1,62 @@ +import type { + ProcessOwnershipHost, + ProcessRunTokenObservation, + ProcessTableEntry, +} from "./ProcessOwnership.js"; + +const errorMessage = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + +/** Read only ownership tokens, using per-process fallback for missing batch rows. */ +export async function* readProcessRunTokens( + host: ProcessOwnershipHost, + processes: readonly ProcessTableEntry[], + signal?: AbortSignal, +): AsyncGenerator<{ + readonly process: ProcessTableEntry; + readonly observation: ProcessRunTokenObservation; +}> { + signal?.throwIfAborted(); + if (processes.length === 0) return; + let batch: ReadonlyMap | undefined; + let batchFailure: string | undefined; + if (host.runTokens !== undefined) { + try { + batch = + signal === undefined + ? await host.runTokens(processes) + : await host.runTokens(processes, signal); + } catch (cause: unknown) { + signal?.throwIfAborted(); + batchFailure = errorMessage(cause); + } + } + for (const process of processes) { + signal?.throwIfAborted(); + let observation = batch?.get(process.pid); + if (observation === undefined) { + try { + const environment = + signal === undefined + ? await host.environment(process.pid) + : await host.environment(process.pid, signal); + observation = { + state: "readable", + runId: environment.REA_PROCESS_RUN_ID, + }; + } catch (cause: unknown) { + signal?.throwIfAborted(); + const reason = errorMessage(cause); + observation = { + state: "unavailable", + reason: + batchFailure === undefined + ? reason + : `${reason}; run-token batch failed: ${batchFailure}`, + }; + } + } + signal?.throwIfAborted(); + yield { process, observation }; + } +} diff --git a/src/process/ProviderDiagnosticRedaction.ts b/src/process/ProviderDiagnosticRedaction.ts new file mode 100644 index 0000000000..ac32b290af --- /dev/null +++ b/src/process/ProviderDiagnosticRedaction.ts @@ -0,0 +1,31 @@ +import type { ProviderProcessSnapshot } from "./ProviderProcess.js"; + +/** Redact explicit transport credentials from complete and budget-cut output. */ +export const redactCapturedTransportCredential = ( + stream: Pick< + ProviderProcessSnapshot["stdout"], + "text" | "bytes" | "observedBytes" + >, + token: string | undefined, + replacement: string, +): string => { + if (token === undefined || token.length === 0) return stream.text; + + const { text } = stream; + const cutOff = stream.observedBytes > stream.bytes; + if (cutOff && !text.endsWith(token)) { + const maximumPrefixLength = Math.min(text.length, token.length - 1); + for ( + let prefixLength = maximumPrefixLength; + prefixLength > 0; + prefixLength -= 1 + ) { + if (text.endsWith(token.slice(0, prefixLength))) + return `${text + .slice(0, text.length - prefixLength) + .replaceAll(token, replacement)}${replacement}`; + } + } + + return text.replaceAll(token, replacement); +}; diff --git a/src/process/ProviderProcess.ts b/src/process/ProviderProcess.ts index 151280b4ff..28631c47cd 100644 --- a/src/process/ProviderProcess.ts +++ b/src/process/ProviderProcess.ts @@ -76,11 +76,16 @@ export interface ProviderProcessSnapshot { readonly diagnosticTruncated?: boolean; readonly stdout: { readonly text: string; + /** Bytes retained in text, limited by maxDiagnosticBytes when configured. */ readonly bytes: number; + /** All bytes drained from this stream since construction or resetOutput. */ + readonly observedBytes: number; }; readonly stderr: { readonly text: string; + /** Bytes retained in text, limited by maxDiagnosticBytes when configured. */ readonly bytes: number; + readonly observedBytes: number; }; readonly exitCode: number | null | undefined; readonly signal: NodeJS.Signals | null | undefined; @@ -268,8 +273,8 @@ export class ProviderProcessSupervisor { snapshot(): ProviderProcessSnapshot { return { ...(this.#diagnosticTruncated ? { diagnosticTruncated: true } : {}), - stdout: this.#stdout.snapshot(), - stderr: this.#stderr.snapshot(), + stdout: this.#stdout.snapshot(this.#stdoutObservedBytes), + stderr: this.#stderr.snapshot(this.#stderrObservedBytes), exitCode: this.#exitObservation?.code, signal: this.#exitObservation?.signal, }; @@ -488,11 +493,12 @@ class ProcessOutputCapture { this.#chunks.push(Buffer.from(chunk)); } - snapshot(): ProviderProcessSnapshot["stdout"] { + snapshot(observedBytes: number): ProviderProcessSnapshot["stdout"] { const text = Buffer.concat(this.#chunks, this.#bytes).toString("utf8"); return { text, bytes: this.#bytes, + observedBytes, }; } } diff --git a/src/process/capture/ProcessCaptureLifecycle.ts b/src/process/capture/ProcessCaptureLifecycle.ts index 19ac0173a9..e4e59d1716 100644 --- a/src/process/capture/ProcessCaptureLifecycle.ts +++ b/src/process/capture/ProcessCaptureLifecycle.ts @@ -25,7 +25,7 @@ import { type ProcessOwnershipBaseline, } from "../ProcessOwnership.js"; import { - observeOwnedProcessGroup, + observeOwnedProcessGroups, prepareProcessOwnershipInspection, systemProcessOwnershipHost, } from "../ProcessOwnershipObservation.js"; @@ -616,21 +616,20 @@ export const observeSettlement = async ( while (!deadlineReached) { onLiveProgress?.(); assertNotCancelled(signal); - const observations = await Promise.all( - [...new Set(processGroupIds)].map((processGroupId) => - observeOwnedProcessGroup( - { runId, leaderPid: processGroupId, processGroupId }, - undefined, - signal, - ), - ), + const observations = await observeOwnedProcessGroups( + runId, + processGroupIds, + undefined, + signal, ); assertNotCancelled(signal); - if (observations.some(({ state }) => state === "unverifiable")) { + if ( + [...observations.values()].some(({ state }) => state === "unverifiable") + ) { recordEvent("lifecycle", 1); return { state: "unverifiable", elapsed_ms: Date.now() - started }; } - if (observations.every(({ state }) => state === "empty")) { + if ([...observations.values()].every(({ state }) => state === "empty")) { consecutiveEmpty += 1; if (consecutiveEmpty >= 2) { recordEvent("lifecycle", 1); diff --git a/tests/boundary/android/androidTargetSnapshot.test.ts b/tests/boundary/android/androidTargetSnapshot.test.ts new file mode 100644 index 0000000000..45d70569fb --- /dev/null +++ b/tests/boundary/android/androidTargetSnapshot.test.ts @@ -0,0 +1,47 @@ +import { stat, truncate, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; + +import { snapshotAndroidTarget } from "../../../src/android/AndroidTargetSnapshot.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +it.skipIf(process.platform === "win32")( + "cancels an in-flight APK snapshot and removes its partial output", + async () => { + const root = await createTestTempDirectory("rea-android-snapshot-cancel-"); + const source = join(root, "large.apk"); + const snapshot = join(root, "target.apk"); + await writeFile(source, ""); + // A sparse extent keeps fixture setup cheap while ensuring the copy cannot + // finish before the test observes its first output bytes and aborts it. + await truncate(source, 64 * 1024 * 1024 * 1024); + + const controller = new AbortController(); + const pending = snapshotAndroidTarget( + source, + "0".repeat(64), + root, + "inspect_android_package", + controller.signal, + ); + try { + await expect + .poll( + () => + stat(snapshot).then( + ({ size }) => size > 0, + () => false, + ), + { timeout: 10_000, interval: 10 }, + ) + .toBe(true); + controller.abort(); + await expect(pending).rejects.toMatchObject({ name: "AbortError" }); + await expect(stat(snapshot)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { + controller.abort(); + await pending.catch(() => undefined); + } + }, +); diff --git a/tests/boundary/android/jadxConfiguration.test.ts b/tests/boundary/android/jadxConfiguration.test.ts index e29bcee495..57beee669d 100644 --- a/tests/boundary/android/jadxConfiguration.test.ts +++ b/tests/boundary/android/jadxConfiguration.test.ts @@ -5,7 +5,8 @@ import { expect, it as test } from "vitest"; import { inspectJadxAvailability, - resolveJadxConfiguration, + resolveJadxConfigurationInputs, + validateJadxConfiguration, } from "../../../src/android/JadxConfiguration.js"; import { AnalysisCapabilityUnavailableError } from "../../../src/domain/analysisErrorCore.js"; import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; @@ -14,6 +15,18 @@ import { writeJadxJarInventory } from "../../fixtures/android/jadxJar.js"; const it = test.skipIf(process.platform === "win32"); +const resolveConfiguration = async ( + environment: Parameters[0], + operation: Parameters[1], +) => { + const configuration = await resolveJadxConfigurationInputs( + environment, + operation, + ); + await validateJadxConfiguration(configuration, environment, operation); + return configuration; +}; + it("rejects malformed or unrelated engine archives with JAR-specific recovery", async () => { const root = await createTestTempDirectory("rea-jadx-invalid-jar-"); const javaHome = await createFakeJdk(root); @@ -29,7 +42,7 @@ it("rejects malformed or unrelated engine archives with JAR-specific recovery", diagnostics: { jar_path: await realpath(jar), phase: "jar-inspection" }, }); await expect( - resolveJadxConfiguration(environment, "inspect_android_package"), + resolveConfiguration(environment, "inspect_android_package"), ).rejects.toMatchObject({ _tag: "AnalysisCapabilityUnavailableError", reason: expect.stringContaining("REA_JADX_MCP_JAR"), @@ -45,7 +58,7 @@ it("rejects an explicit JAVA_HOME without an executable java binary", async () = await writeJadxJarInventory(jar); await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome }, "inspect_android_package", ), @@ -62,7 +75,7 @@ it("accepts a full JDK and canonicalizes the selected JAR", async () => { await writeJadxJarInventory(jar); await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome }, "inspect_android_package", ), @@ -78,7 +91,7 @@ it("configures heap and visible processors independently without forcing either" const jar = join(root, "engine.jar"); const javaHome = await createFakeJdk(root); await writeJadxJarInventory(jar); - const configuration = await resolveJadxConfiguration( + const configuration = await resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome, @@ -92,7 +105,7 @@ it("configures heap and visible processors independently without forcing either" "-XX:ActiveProcessorCount=4", ]); await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome, @@ -107,7 +120,7 @@ it("configures heap and visible processors independently without forcing either" for (const name of ["REA_JADX_HEAP_MIB", "REA_JADX_ACTIVE_PROCESSOR_COUNT"]) for (const value of ["", "0", "-1", "1.5", "8g", " 4", "9007199254740992"]) await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome, @@ -127,7 +140,7 @@ it("distinguishes a runnable JRE from a missing Java executable", async () => { await writeJadxJarInventory(jar); const jreHome = await createFakeJdk(root, false); await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: jreHome }, "inspect_android_package", ), @@ -175,7 +188,7 @@ it("preserves exact Java probe output and marks max-buffer capture as truncated" ); await chmod(java, 0o700); - const failure = await resolveJadxConfiguration( + const failure = await resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: javaHome }, "inspect_android_package", ).then( @@ -241,7 +254,7 @@ it("distinguishes an unreported Java version from an unsupported old version", a ); await chmod(join(oldHome, "bin", "java"), 0o700); await expect( - resolveJadxConfiguration( + resolveConfiguration( { REA_JADX_MCP_JAR: jar, JAVA_HOME: oldHome }, "inspect_android_package", ), diff --git a/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts b/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts index b74c600815..5fd1e061bb 100644 --- a/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts +++ b/tests/boundary/browser/cdpBrowserProvider.document_script.test.ts @@ -294,6 +294,15 @@ describe("CdpBrowserProvider: WebMCP discovery and completeness", () => { ], }, }); + expect(result.value.completeness.excluded).toContainEqual( + expect.objectContaining({ + section: "webmcp_tools", + reason: "out_of_target_scope", + }), + ); + expect(result.value.completeness.unavailable_sections).not.toContain( + "webmcp_tools", + ); const serialized = JSON.stringify(result.value); expect(serialized).toContain("tool-secret"); expect(serialized).toContain("tool-source-secret"); @@ -354,7 +363,9 @@ describe("CdpBrowserProvider: WebMCP discovery and completeness", () => { "WebMCP.invokeTool", ); }); +}); +describe("CdpBrowserProvider: WebMCP replay inventory", () => { it("returns every WebMCP registration in the replay", async () => { const browser = await startFakeCdpBrowser({ webMcpTools: true, @@ -410,7 +421,98 @@ describe("CdpBrowserProvider: WebMCP discovery and completeness", () => { }); }); -describe("CdpBrowserProvider WebMCP inventory completeness", () => { +describe("CdpBrowserProvider WebMCP malformed replay completeness", () => { + it.each(["toolsAdded", "toolsRemoved"] as const)( + "keeps malformed %s replay entries from proving WebMCP equality", + async (malformedEvent) => { + let malformed = false; + const registration = { + name: "read_item", + frameId: "frame-main", + inputSchema: { type: "object" }, + }; + const browser = await startFakeCdpBrowser({ + commandEvents: (command) => { + if (command.method !== "WebMCP.enable") return undefined; + const events: { + readonly method: string; + readonly params: unknown; + readonly sessionId?: string; + }[] = [ + { + method: "WebMCP.toolsAdded", + ...(command.sessionId === undefined + ? {} + : { sessionId: command.sessionId }), + params: { + tools: + malformed && malformedEvent === "toolsAdded" + ? [registration, null] + : [registration], + }, + }, + ]; + if (malformed && malformedEvent === "toolsRemoved") + events.push({ + method: "WebMCP.toolsRemoved", + ...(command.sessionId === undefined + ? {} + : { sessionId: command.sessionId }), + params: { tools: [{ name: registration.name }, null] }, + }); + return events; + }, + }); + trackBrowser(browser); + const provider = new CdpBrowserProvider(); + const input = { + cdp_endpoint: browser.endpoint, + allowed_origins: [browser.allowedOrigin], + target_id: "allowed-page", + observation_ms: 0, + }; + const inspection = await provider.inspectPage( + inspectWebPageInputSchema.parse(input), + ); + if (!inspection.ok) throw inspection.error; + const discover = async () => { + const result = await provider.discoverWebMcpTools( + discoverWebMcpToolsInputSchema.parse(input), + ); + if (!result.ok) throw result.error; + return result.value; + }; + const before = await discover(); + expect(before.tools.items).toHaveLength(1); + expect(before.completeness.unavailable_sections).not.toContain( + "webmcp_tools", + ); + + malformed = true; + const after = await discover(); + expect(after.tools.items).toHaveLength(1); + expect(after.completeness.unavailable_sections).toContain("webmcp_tools"); + expect(after.completeness.excluded).toContainEqual( + expect.objectContaining({ + section: "webmcp_tools", + reason: "invalid_protocol_value", + }), + ); + const comparison = compareWebCaptures( + compareWebCapturesInputSchema.parse({ + before: { inspection: inspection.value, webmcp: before }, + after: { inspection: inspection.value, webmcp: after }, + }), + ); + expect(comparison.dimensions.webmcp).toMatchObject({ + status: "unknown", + total_changes: 0, + }); + }, + ); +}); + +describe("CdpBrowserProvider WebMCP schema comparison", () => { it("compares full untrusted declarations independently of object-key order", async () => { const declaration: JsonValue = { type: "object", diff --git a/tests/boundary/browser/cdpCleanupCancellation.test.ts b/tests/boundary/browser/cdpCleanupCancellation.test.ts index e9508297b5..0d4c17a0fa 100644 --- a/tests/boundary/browser/cdpCleanupCancellation.test.ts +++ b/tests/boundary/browser/cdpCleanupCancellation.test.ts @@ -47,6 +47,7 @@ it.each(["Page.disable", "Target.detachFromTarget"])( expect(methods).toContain("Target.detachFromTarget"); expect(methods).not.toContain("Target.closeTarget"); expect(methods).not.toContain("Browser.close"); + await vi.waitFor(() => expect(browser.activeConnectionCount()).toBe(0)); }, ); @@ -76,6 +77,11 @@ it("releases cleanup when cancellation arrives after capture completes", async ( expect(browser.commands.map(({ method }) => method)).toContain( "Target.detachFromTarget", ); + await vi.waitFor(() => expect(browser.activeConnectionCount()).toBe(0)); + const methods = browser.commands.map(({ method }) => method); + expect(methods).not.toContain("Target.closeTarget"); + expect(methods).not.toContain("Browser.close"); + expect((await fetch(`${browser.endpoint}/json/version`)).ok).toBe(true); }); it("closes an Electron inspection session when cleanup is cancelled", async () => { @@ -122,11 +128,11 @@ it("closes an Electron inspection session when cleanup is cancelled", async () = operation: "inspect_electron_page", }, }); - expect(browser.commands.map(({ method }) => method)).toContain( - "Page.disable", - ); const methods = browser.commands.map(({ method }) => method); + expect(methods).not.toContain("Page.disable"); expect(methods).toContain("Target.detachFromTarget"); expect(methods).not.toContain("Target.closeTarget"); expect(methods).not.toContain("Browser.close"); + await vi.waitFor(() => expect(browser.activeConnectionCount()).toBe(0)); + expect((await fetch(`${browser.endpoint}/json/version`)).ok).toBe(true); }); diff --git a/tests/boundary/browser/cdpJsonShapeComparison.test.ts b/tests/boundary/browser/cdpJsonShapeComparison.test.ts index e6dfa8ef78..6784b478e9 100644 --- a/tests/boundary/browser/cdpJsonShapeComparison.test.ts +++ b/tests/boundary/browser/cdpJsonShapeComparison.test.ts @@ -39,6 +39,68 @@ const equivalentBodies = [ }, ]; +it("captures all repeated array rows and empty arrays without retaining payload values", async () => { + const count = 6_000; + const body = JSON.stringify( + Array.from({ length: count }, (_, index) => ({ + value: index % 2 === 0 ? index : "payload-value-excluded", + nested: index % 2 === 0 ? [] : [true, null], + empty: [], + "*": { "": "payload-value-excluded" }, + })), + ); + const before = captureBody("base64 response", body); + const shape = before.network.requests[0]?.body_shapes.response; + const element = { kind: "array-element" }; + const property = (name: string) => ({ kind: "property", name }); + expect(shape).toMatchObject({ + root_type: "array", + node_count: count * 7 + 1, + max_depth_observed: 3, + properties: [ + { path: [element], types: ["object"], observations: count }, + { + path: [element, property("*")], + types: ["object"], + observations: count, + }, + { + path: [element, property("*"), property("")], + types: ["string"], + observations: count, + }, + { + path: [element, property("empty")], + types: ["array"], + observations: count, + }, + { + path: [element, property("nested")], + types: ["array"], + observations: count, + }, + { + path: [element, property("nested"), element], + types: ["boolean", "null"], + observations: count, + }, + { + path: [element, property("value")], + types: ["number", "string"], + observations: count, + }, + ], + }); + expect(JSON.stringify(shape)).not.toContain("payload-value-excluded"); + const after = captureBody("base64 response", body); + expect( + (await compareCaptures(before, after)).dimensions.network, + ).toMatchObject({ + status: "unknown", + total_changes: 0, + }); +}); + describe.each(["request", "response", "base64 response"])( "CDP %s JSON shape comparison", (source) => { diff --git a/tests/boundary/browser/cdpWebRuntime.test.ts b/tests/boundary/browser/cdpWebRuntime.test.ts index ac9049c626..7b5c772ca4 100644 --- a/tests/boundary/browser/cdpWebRuntime.test.ts +++ b/tests/boundary/browser/cdpWebRuntime.test.ts @@ -188,6 +188,107 @@ describe("runtime window and producer granularity", () => { }); }); +it("joins a reordered multi-script coverage sample by identity and preserves unavailable bounds", async () => { + const texts = new Map( + Array.from({ length: 256 }, (_, index) => [ + `script-${index}`, + `// 🔥\r\n${" ".repeat(index)}function f(){return ${index};}`, + ]), + ); + const { browser, provider, execution } = await fixture({ + commandEvents: (command, origin) => + command.method === "Debugger.enable" + ? [...texts.keys(), "foreign-script"].map((scriptId) => ({ + ...(command.sessionId === undefined + ? {} + : { sessionId: command.sessionId }), + method: "Debugger.scriptParsed", + params: { + scriptId, + url: `${origin}/shared.js`, + executionContextId: scriptId === "foreign-script" ? 99 : 1, + startLine: 0, + startColumn: 0, + endLine: 1, + endColumn: 0, + }, + })) + : undefined, + commandError: (command) => + command.method === "Debugger.getScriptSource" && + command.params.scriptId === "script-128" + ? { code: -32000, message: "Selected source is unavailable" } + : undefined, + commandResult: (command, origin) => { + if (command.method === "Debugger.getScriptSource") + return { scriptSource: texts.get(String(command.params.scriptId)) }; + if (command.method !== "Profiler.takePreciseCoverage") return undefined; + return { + timestamp: 13, + result: [...texts] + .reverse() + .map(([scriptId, text]) => ({ + scriptId, + url: `${origin}/shared.js`, + functions: [ + { + functionName: "f", + isBlockCoverage: true, + ranges: [ + { startOffset: 0, endOffset: text.length, count: 1 }, + { startOffset: 8, endOffset: 9, count: 0 }, + ], + }, + ], + })) + .concat([ + { + scriptId: "foreign-script", + url: `${origin}/shared.js`, + functions: [], + }, + ]), + }; + }, + }); + const result = await provider.observeExecution(execution); + if (!result.ok) throw result.error; + expect(result.value.coverage.excluded_scripts).toBe(1); + expect( + result.value.coverage.scripts.map(({ script_id }) => script_id), + ).toEqual([...texts.keys()].reverse()); + for (const script of result.value.coverage.scripts) { + expect(script.functions[0]?.ranges).toEqual([ + { + start_offset: 0, + end_offset: texts.get(script.script_id)?.length, + count: 1, + source_bounds: + script.script_id === "script-128" ? "unknown" : "verified", + }, + { + start_offset: 8, + end_offset: 9, + count: 0, + source_bounds: + script.script_id === "script-128" ? "unknown" : "verified", + }, + ]); + } + expect( + result.value.sources.find(({ script_id }) => script_id === "script-128") + ?.source, + ).toMatchObject({ state: "unavailable" }); + expect( + browser.commands.some( + ({ method }) => method === "Profiler.stopPreciseCoverage", + ), + ).toBe(true); + expect( + browser.commands.some(({ method }) => method === "Target.detachFromTarget"), + ).toBe(true); +}); + describe("runtime source and listener identity", () => { it("retains same-URL script identities, UTF-16 bounds, nested zero counts and actual cleanup", async () => { const { browser, provider, execution } = await fixture(); diff --git a/tests/boundary/browser/cdpWebRuntimeSourceIdentity.test.ts b/tests/boundary/browser/cdpWebRuntimeSourceIdentity.test.ts index e14b81d207..ef97f900ad 100644 --- a/tests/boundary/browser/cdpWebRuntimeSourceIdentity.test.ts +++ b/tests/boundary/browser/cdpWebRuntimeSourceIdentity.test.ts @@ -5,6 +5,59 @@ import { observeWebExecutionInputSchema } from "../../../src/domain/webExecution import { inspectWebEventListenersInputSchema } from "../../../src/domain/webEventListeners.js"; import { startRuntimeBrowser } from "../../fixtures/webRuntime.js"; +it("invalidates retained sources when their execution context is reused during another source read", async () => { + let emit: ((event: CdpEvent) => void) | undefined; + const browser = await startRuntimeBrowser({ + commandResult: (command, origin) => { + if ( + command.method === "Debugger.getScriptSource" && + command.params.scriptId === "script-b" + ) + emit?.({ + sessionId: "session-1", + method: "Runtime.executionContextCreated", + params: { + context: { + id: 1, + uniqueId: "replacement-context", + origin, + name: "", + auxData: { + frameId: "runtime-main", + isDefault: true, + type: "default", + }, + }, + }, + }); + return undefined; + }, + }); + emit = browser.emitEvent; + onTestFinished(() => browser.close()); + const result = await new CdpWebRuntimeProvider().observeExecution( + observeWebExecutionInputSchema.parse({ + cdp_endpoint: browser.endpoint, + target_id: "allowed-page", + observation_ms: 5, + }), + ); + if (!result.ok) throw result.error; + expect(result.value.coverage.scripts).toEqual([]); + expect(result.value.coverage.excluded_scripts).toBe(3); + expect(result.value.script_inventory.main_document_scripts).toBe(0); + for (const scriptId of ["script-a", "script-b"]) + expect( + result.value.sources.find(({ script_id }) => script_id === scriptId), + ).toMatchObject({ frame_id: null, source: { state: "excluded" } }); + expect(browser.commands.map(({ method }) => method)).toEqual( + expect.arrayContaining([ + "Profiler.stopPreciseCoverage", + "Target.detachFromTarget", + ]), + ); +}); + it.each( ["listeners", "execution"].flatMap((operation) => ["source_read", "other_source_read", "final_assertion"].map((phase) => ({ diff --git a/tests/boundary/cli/dylibResolutionCli.test.ts b/tests/boundary/cli/dylibResolutionCli.test.ts index c92d265b29..1f64499935 100644 --- a/tests/boundary/cli/dylibResolutionCli.test.ts +++ b/tests/boundary/cli/dylibResolutionCli.test.ts @@ -252,6 +252,65 @@ describe("trace-dylib-resolution CLI", () => { ); }); +cliTest( + "retains directory requirements when resolving dylib install names", + async ({ cli }) => { + const directory = await createTestTempDirectory("rea-dylib-path-cli-"); + const names = [ + "@loader_path/lib/libhelper.dylib/", + "@loader_path/lib/libhelper.dylib/.", + "@rpath/libhelper.dylib/", + "@rpath/libhelper.dylib/.", + "@rpath/", + "@loader_path/lib/./libhelper.dylib", + "@loader_path/lib//libhelper.dylib", + ]; + await writeFiles(directory, { + tool: machoImage({ + commands: [ + buildVersionCommand(1), + rpathCommand("@loader_path/lib/libhelper.dylib"), + rpathCommand("@loader_path/lib"), + ...names.map((name) => dylibCommand(LC.LOAD_DYLIB, name)), + ], + }), + "lib/libhelper.dylib": machoImage({ + fileType: FILE_TYPE.dylib, + commands: [buildVersionCommand(1)], + }), + }); + const result = await cli.run({ + arguments: ["trace-dylib-resolution", join(directory, "tool"), "--json"], + environment: ENVIRONMENT, + }); + expect(result.exitCode, JSON.stringify(result.json)).toBe(0); + const trace = dylibResolutionResultSchema.parse( + parseEvidence(result.json).normalized_result, + ); + expect( + trace.edges.map(({ install_name, candidates, resolution }) => ({ + install_name, + outcome: candidates[0]?.outcome, + resolution, + })), + ).toEqual( + names.map((name, index) => ({ + install_name: name, + outcome: index < 5 ? "absent" : "resolved", + resolution: + index < 5 + ? { status: "unresolved", image: null } + : { status: "resolved", image: "lib/libhelper.dylib" }, + })), + ); + expect( + trace.findings.map(({ kind, edge_index }) => [kind, edge_index]), + ).toEqual( + [0, 1, 2, 3, 4].map((index) => ["required-load-unresolved", index]), + ); + }, +); + describe.skipIf(process.getuid?.() === 0)( "trace-dylib-resolution permissions", () => { diff --git a/tests/boundary/cli/evidenceBundleCommands.test.ts b/tests/boundary/cli/evidenceBundleCommands.test.ts new file mode 100644 index 0000000000..418c20eb49 --- /dev/null +++ b/tests/boundary/cli/evidenceBundleCommands.test.ts @@ -0,0 +1,96 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect } from "vitest"; + +import { createEvidence } from "../../../src/domain/evidence.js"; +import { createEvidenceBundle } from "../../../src/domain/evidenceBundle.js"; +import { + createResidualUnknown, + updateResidualUnknown, +} from "../../../src/domain/residualUnknown.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +const revisionBundle = () => { + const records = ["open", "investigate"].map((operation) => + createEvidence( + undefined, + { id: "fixture", name: "Fixture", version: "1" }, + { operation, parameters: {}, result: { observed: operation } }, + ), + ); + const [opened, investigated] = records; + if (opened === undefined || investigated === undefined) + throw new Error("Missing mutation Evidence"); + const details = { + severity: "high" as const, + supporting_evidence_ids: [], + contradicting_evidence_ids: [], + required_authority: null, + required_confidence: "observed" as const, + required_environment: null, + recommended_probes: [], + relationships: [], + }; + const initial = createResidualUnknown( + { question: "How is this input handled?", domain: "protocol", ...details }, + opened.evidence_id, + null, + ); + const revised = updateResidualUnknown( + initial, + { + unknown_id: initial.unknown_id, + expected_revision: initial.revision, + status: "investigating", + ...details, + resolution: null, + }, + investigated.evidence_id, + ); + return createEvidenceBundle(records, [initial, revised]); +}; + +cliTest( + "reports standalone import counts for every unknown revision", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-evidence-import-"); + const path = join(root, "bundle.json"); + await writeFile(path, JSON.stringify(revisionBundle())); + const result = await cli.run({ + arguments: ["evidence-import", path, "--json"], + }); + expect(result.exitCode).toBe(0); + expect(result.json).toEqual({ imported: 2, unknowns_added: 2, total: 2 }); + }, +); + +cliTest( + "rejects altered Evidence and incomplete unknown history before reporting counts", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-evidence-import-invalid-"); + const path = join(root, "bundle.json"); + const bundle = revisionBundle(); + for (const invalid of [ + { + ...bundle, + records: bundle.records.map((record) => ({ + ...record, + normalized_result: "altered", + })), + }, + { ...bundle, unknowns: bundle.unknowns.slice(1) }, + ]) { + await writeFile(path, JSON.stringify(invalid)); + const result = await cli.run({ + arguments: ["evidence-import", path, "--json"], + }); + expect(result.exitCode).toBe(1); + expect(result.json).toMatchObject({ + code: "evidence_integrity_mismatch", + }); + expect(result.json).not.toHaveProperty("imported"); + } + }, +); diff --git a/tests/boundary/evm/evmoleInterfaceProvider.test.ts b/tests/boundary/evm/evmoleInterfaceProvider.test.ts index 5d1305d21b..8956c5b0f8 100644 --- a/tests/boundary/evm/evmoleInterfaceProvider.test.ts +++ b/tests/boundary/evm/evmoleInterfaceProvider.test.ts @@ -7,6 +7,7 @@ import { projectAnalysisError } from "../../../src/domain/analysisErrorProjectio import { evmInterfaceSchema } from "../../../src/domain/evm/evmInterface.js"; import { EvmoleInterfaceProvider } from "../../../src/evm/EvmoleInterfaceProvider.js"; import { spawnOwnedProviderProcess } from "../../../src/process/ProviderProcess.js"; +import { cleanupOwnedProcessGroup } from "../../../src/process/ProcessOwnership.js"; import { waitForProviderProcessReady } from "../../fixtures/providerProcess.js"; import { PrivateRuntimeRoot } from "../../../src/process/PrivateRuntimeRoot.js"; import { @@ -128,31 +129,75 @@ it.runIf(!unsupportedHost)( async () => { const { path } = await fixture(); let ownedPath = ""; + let failCleanup = true; + let launches = 0; const provider = new EvmoleInterfaceProvider({}, async (spawn) => { + launches += 1; ownedPath = spawn.cwd ?? ""; const launched = await spawnOwnedProviderProcess({ ...spawn, command: process.execPath, arguments: ["-e", 'process.stdout.write("x".repeat(1048577))'], }); + const cleanupOwned = + launched.cleanup ?? + (() => cleanupOwnedProcessGroup(launched.ownership)); return { ...launched, cleanup: async () => { - await launched.cleanup?.(); - throw new Error("injected post-cleanup reporting failure"); + const cleaned = await cleanupOwned(); + if (failCleanup) + throw new Error("injected post-cleanup reporting failure"); + return cleaned; }, }; }); + onTestFinished(async () => { + failCleanup = false; + await provider.close(); + }); const result = await provider.inspect({ path, encoding: "hex" }); if (result.ok) throw new Error("Expected compounded lifecycle failure"); - expect(result.error).toMatchObject({ - cleanupIncomplete: true, - diagnostics: { - reason: "injected post-cleanup reporting failure", - previous_error: { failure_kind: "output-limit" }, - captured_output: { truncated: true, stderr: "" }, + expect(result.error.cleanupIncomplete).toBe(true); + // Inspect the retained error chain without diffing a megabyte of stdout. + const cleanup = z + .object({ + reason: z.string(), + previous_error: z + .object({ + failure_kind: z.string(), + stdout: z.unknown().optional(), + stderr: z.unknown().optional(), + }) + .transform(({ failure_kind, stdout, stderr }) => ({ + failure_kind, + output_repeated: stdout !== undefined || stderr !== undefined, + })), + captured_output: z.object({ + truncated: z.boolean(), + stderr: z.literal(""), + stdout: z.string().transform((value) => Buffer.byteLength(value)), + }), + }) + .parse(projectAnalysisError(result.error).details?.["diagnostics"]); + expect(cleanup).toEqual({ + reason: "injected post-cleanup reporting failure", + previous_error: { + failure_kind: "output-limit", + output_repeated: false, }, + captured_output: { truncated: true, stderr: "", stdout: 1048576 }, }); + expect(result.error.cleanupResources).toContain(ownedPath); + await access(ownedPath); + const blocked = await provider.inspect({ path, encoding: "hex" }); + expect(blocked.ok).toBe(false); + expect(launches).toBe(1); + await expect(provider.close()).rejects.toMatchObject({ + cleanupIncomplete: true, + }); + failCleanup = false; + await provider.close(); await expect(access(ownedPath)).rejects.toMatchObject({ code: "ENOENT" }); }, ); diff --git a/tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts b/tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts index 4d40227f32..04cb7b7079 100644 --- a/tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts +++ b/tests/boundary/filesystem/artifactInventoryEntryOrder.test.ts @@ -69,6 +69,35 @@ describe("artifact inventory entry order", () => { }, ); + it("resolves deep ZIP members without inventing missing directories", async () => { + const root = await createTestTempDirectory("rea-deep-archive-parent-"); + const archive = join(root, "deep.zip"); + // Fits the ZIP filename field while exceeding the formerly expensive depth. + const members = ["data.txt", "second.txt", "third.bin"].map( + (name) => `observed/${"d/".repeat(30_000)}${name}`, + ); + await writeOrderedZip(archive, [...members, "observed/", "unrelated/"]); + const inventory = await inventoryArtifact(archive); + expect(artifactParentPaths(inventory)).toEqual({ + ".": null, + observed: ".", + unrelated: ".", + ...Object.fromEntries(members.map((member) => [member, "observed"])), + }); + const observedParent = artifactOccurrenceAt(inventory, "observed"); + for (const member of members) { + const occurrence = artifactOccurrenceAt(inventory, member); + expect(occurrence.hash_status).toBe("verified"); + expect(inventory.edges).toContainEqual( + expect.objectContaining({ + occurrence_id: occurrence.occurrence_id, + parent_artifact_id: observedParent.artifact_id, + relation: "contains", + }), + ); + } + }); + it.each([ { entries: ["pkg/sub/data.txt"], @@ -129,6 +158,61 @@ describe("entry-order validation and reader controls", () => { } }); + it("retains high-cardinality case-folded directories and their descendants", async () => { + const root = await createTestTempDirectory( + "rea-entry-order-case-capacity-", + ); + const archive = join(root, "case-capacity.zip"); + const variants = Array.from({ length: 256 }, (_, mask) => + [..."abcdefgh"] + .map((character, index) => + (mask & (1 << index)) === 0 ? character : character.toUpperCase(), + ) + .join(""), + ); + await writeOrderedZip( + archive, + variants.flatMap((variant) => [variant + "/", variant + "/entry.txt"]), + ); + + const inventory = await inventoryArtifact(archive); + const directories = variants.map((variant) => + artifactOccurrenceAt(inventory, variant), + ); + const descendants = variants.map((variant) => + artifactOccurrenceAt(inventory, `${variant}/entry.txt`), + ); + const ordered = [...variants].sort(); + const first = ordered[0]; + const second = ordered[1]; + + expect(inventory.occurrences).toHaveLength(1 + variants.length * 2); + expect( + new Set(inventory.occurrences.map(({ logical_path }) => logical_path)) + .size, + ).toBe(1 + variants.length * 2); + expect( + new Set( + [...directories, ...descendants].map( + ({ occurrence_id }) => occurrence_id, + ), + ).size, + ).toBe(variants.length * 2); + expect( + directories.every(({ limitations }) => limitations.length === 1), + ).toBe(true); + expect( + descendants.every(({ limitations }) => limitations.length === 1), + ).toBe(true); + expect( + directories.find(({ logical_path }) => logical_path === first) + ?.limitations[0], + ).toContain(`with ${second} and 254 other spellings;`); + expect(descendants[0]?.limitations[0]).toContain( + `is under ${variants[0]}, which collides under English (en-US) Unicode case folding with ${first} and 254 other spellings;`, + ); + }); + it("preserves ordinary directory and ASAR structure", async () => { const root = await createTestTempDirectory("rea-entry-order-readers-"); const directory = join(root, "source"); diff --git a/tests/boundary/filesystem/asarStreamingAllocation.test.ts b/tests/boundary/filesystem/asarStreamingAllocation.test.ts index cc94381797..24584063d2 100644 --- a/tests/boundary/filesystem/asarStreamingAllocation.test.ts +++ b/tests/boundary/filesystem/asarStreamingAllocation.test.ts @@ -305,19 +305,22 @@ describe("ASAR read failures", () => { errno: -5, syscall: "fstat", }); + const closeError = Object.assign(new Error("device close failed"), { + code: "EIO", + errno: -5, + syscall: "close", + }); let closeCount = 0; const reader = new AsarArtifactReader(archive, async (path, flags) => { const handle = await openFile(path, flags); expect(await matchingDescriptorCount(identity)).toBeGreaterThan(0); - let statCount = 0; return { async stat() { - statCount += 1; - if (statCount === 2) throw ioError; - return handle.stat(); + throw ioError; }, async close() { closeCount += 1; + if (closeCount === 1) throw closeError; await handle.close(); }, createReadStream: (options) => handle.createReadStream(options), @@ -337,73 +340,320 @@ describe("ASAR read failures", () => { message: expect.stringContaining( `Could not read ${entry.path} ASAR at ${archive}: device stat failed`, ), - cause: { code: "EIO", errno: -5, syscall: "fstat" }, + cause: { + reason: "io", + cause: { code: "EIO", errno: -5, syscall: "fstat" }, + }, + cleanup: { + resources: [unpackedPath], + reason: "device close failed", + }, }); expect(closeCount).toBe(1); + expect(await matchingDescriptorCount(identity)).toBeGreaterThan(0); + await reader.close(); + expect(closeCount).toBe(2); await waitForNoMatchingDescriptor(identity); } finally { await reader.close(); } }, ); +}); - it.each(["missing", "replaced"] as const)( - "reads a captured zero-length member when the original container is %s", - async (change) => { - const root = await createTestTempDirectory("rea-asar-empty-member-"); - const source = join(root, "source"); - const archive = join(root, "fixture.asar"); - await mkdir(source); - await writeFile(join(source, "empty.js"), ""); - await createPackageWithOptions(source, archive, {}); - const reader = new AsarArtifactReader(archive); +it.skipIf(process.platform === "win32")( + "retries cleanup when cancellation arrives immediately after unpacked stat", + async () => { + const root = await createTestTempDirectory("rea-asar-unpacked-abort-stat-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "small.js"), "module.exports = 1;\n"); + await createPackageWithOptions(source, archive, { unpack: "small.js" }); + const unpackedPath = join(`${archive}.unpacked`, "small.js"); + const identity = await lstat(unpackedPath); + const controller = new AbortController(); + let closeCount = 0; + const reader = new AsarArtifactReader(archive, async (path, flags) => { + const handle = await openFile(path, flags); + return { + async stat() { + const observed = await handle.stat(); + controller.abort(); + return observed; + }, + async close() { + closeCount += 1; + if (closeCount === 1) throw new Error("first close failed"); + await handle.close(); + }, + createReadStream: (options) => handle.createReadStream(options), + }; + }); - try { - const entries = []; - for await (const entry of reader.entries()) entries.push(entry); - const entry = entries.find(({ path }) => path === "empty.js"); - if (entry === undefined) throw new Error("Expected empty ASAR member"); - if (change === "missing") await rm(archive); - else await writeFile(archive, "replacement container"); + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries.find(({ path }) => path === "small.js"); + if (entry === undefined) throw new Error("Expected unpacked ASAR member"); - expect(await buffer(await reader.open(entry))).toHaveLength(0); - } finally { - await reader.close(); - } - }, + await expect(reader.open(entry, controller.signal)).rejects.toMatchObject( + { + reason: "cancelled", + cleanup: { + resources: [unpackedPath], + reason: "first close failed", + }, + }, + ); + expect(closeCount).toBe(1); + expect(await matchingDescriptorCount(identity)).toBeGreaterThan(0); + await reader.close(); + expect(closeCount).toBe(2); + await waitForNoMatchingDescriptor(identity); + } finally { + await reader.close(); + } + }, +); + +it.skipIf(process.platform === "win32")( + "preserves an unpacked identity failure when closing its handle fails", + async () => { + const root = await createTestTempDirectory("rea-asar-unpacked-identity-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "small.js"), "module.exports = 1;\n"); + await createPackageWithOptions(source, archive, { unpack: "small.js" }); + const unpackedPath = join(`${archive}.unpacked`, "small.js"); + const identity = await lstat(unpackedPath); + const closeError = new Error("first close failed"); + let closeCount = 0; + const reader = new AsarArtifactReader(archive, async (path, flags) => { + const handle = await openFile(path, flags); + return { + async stat() { + const observed = await handle.stat(); + return Object.assign(Object.create(observed), { + ino: observed.ino + 1, + }); + }, + async close() { + closeCount += 1; + if (closeCount === 1) throw closeError; + await handle.close(); + }, + createReadStream: (options) => handle.createReadStream(options), + }; + }); + + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries.find(({ path }) => path === "small.js"); + if (entry === undefined) throw new Error("Expected unpacked ASAR member"); + + await expect(reader.open(entry)).rejects.toMatchObject({ + name: "ArtifactReaderFailure", + reason: "integrity", + message: expect.stringContaining( + `ASAR unpacked entry changed before read: ${entry.path}`, + ), + cleanup: { + resources: [unpackedPath], + reason: "first close failed", + }, + }); + expect(closeCount).toBe(1); + expect(await matchingDescriptorCount(identity)).toBeGreaterThan(0); + await reader.close(); + expect(closeCount).toBe(2); + await waitForNoMatchingDescriptor(identity); + } finally { + await reader.close(); + } + }, +); + +it.skipIf(process.platform === "win32")( + "retains unpacked handles and snapshots across combined cleanup failures", + async () => { + const root = await createTestTempDirectory("rea-asar-combined-cleanup-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "small.js"), "module.exports = 1;\n"); + await createPackageWithOptions(source, archive, { unpack: "small.js" }); + const unpackedPath = join(`${archive}.unpacked`, "small.js"); + const identity = await lstat(unpackedPath); + const closeError = new Error("unpacked close denied"); + const snapshotError = new Error("snapshot removal denied"); + let closeCount = 0; + const snapshotPaths: string[] = []; + const reader = new AsarArtifactReader( + archive, + async (path, flags) => { + const handle = await openFile(path, flags); + return { + async stat() { + throw Object.assign(new Error("device stat failed"), { + code: "EIO", + errno: -5, + syscall: "fstat", + }); + }, + async close() { + closeCount += 1; + if (closeCount < 3) throw closeError; + await handle.close(); + }, + createReadStream: (options) => handle.createReadStream(options), + }; + }, + async (path) => { + snapshotPaths.push(path); + if (snapshotPaths.length === 1) throw snapshotError; + await rm(path, { recursive: true, force: true }); + }, + ); + + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries.find(({ path }) => path === "small.js"); + if (entry === undefined) throw new Error("Expected unpacked ASAR member"); + + await expect(reader.open(entry)).rejects.toMatchObject({ + reason: "io", + cleanup: { resources: [unpackedPath] }, + }); + expect(closeCount).toBe(1); + + const cleanupFailure = await reader.close().then( + () => undefined, + (cause: unknown) => cause, + ); + expect(cleanupFailure).toMatchObject({ + cleanup: { + resources: [unpackedPath, expect.any(String)], + reason: expect.stringContaining("unpacked close denied"), + }, + }); + const snapshotRoot = snapshotPaths[0]; + if (snapshotRoot === undefined) + throw new Error("Expected owned ASAR snapshot root"); + expect(snapshotPaths).toEqual([snapshotRoot]); + expect(cleanupFailure).toMatchObject({ + cleanup: { resources: [unpackedPath, snapshotRoot] }, + }); + expect(closeCount).toBe(2); + expect(await matchingDescriptorCount(identity)).toBeGreaterThan(0); + expect((await lstat(snapshotRoot)).isDirectory()).toBe(true); + + await reader.close(); + expect(closeCount).toBe(3); + expect(snapshotPaths).toEqual([snapshotRoot, snapshotRoot]); + await waitForNoMatchingDescriptor(identity); + await expect(lstat(snapshotRoot)).rejects.toMatchObject({ + code: "ENOENT", + }); + } finally { + await reader.close(); + } + }, +); + +it.skipIf(process.platform === "win32")( + "releases unpacked handles after stream completion and cancellation", + async () => { + const root = await createTestTempDirectory("rea-asar-unpacked-stream-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "small.js"), "module.exports = 1;\n"); + await createPackageWithOptions(source, archive, { unpack: "small.js" }); + const identity = await lstat(join(`${archive}.unpacked`, "small.js")); + const reader = new AsarArtifactReader(archive); + + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries.find(({ path }) => path === "small.js"); + if (entry === undefined) throw new Error("Expected unpacked ASAR member"); + expect((await buffer(await reader.open(entry))).toString()).toBe( + "module.exports = 1;\n", + ); + await waitForNoMatchingDescriptor(identity); + + const controller = new AbortController(); + const output = await reader.open(entry, controller.signal); + const reading = buffer(output); + controller.abort(); + await expect(reading).rejects.toMatchObject({ reason: "cancelled" }); + await waitForNoMatchingDescriptor(identity); + } finally { + await reader.close(); + } + }, +); + +it.each(["missing", "replaced"] as const)( + "reads a captured zero-length member when the original container is %s", + async (change) => { + const root = await createTestTempDirectory("rea-asar-empty-member-"); + const source = join(root, "source"); + const archive = join(root, "fixture.asar"); + await mkdir(source); + await writeFile(join(source, "empty.js"), ""); + await createPackageWithOptions(source, archive, {}); + const reader = new AsarArtifactReader(archive); + + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries.find(({ path }) => path === "empty.js"); + if (entry === undefined) throw new Error("Expected empty ASAR member"); + if (change === "missing") await rm(archive); + else await writeFile(archive, "replacement container"); + + expect(await buffer(await reader.open(entry))).toHaveLength(0); + } finally { + await reader.close(); + } + }, +); + +it("preserves filesystem errors raised while consuming a member stream", async () => { + const ioError = Object.assign(new Error("device read failed"), { + code: "EIO", + errno: -5, + syscall: "read", + }); + const source = Readable.from( + (async function* () { + yield Buffer.from("partial"); + throw ioError; + })(), + ); + const output = readValidatedAsarEntry( + source, + undefined, + "nested/member.bin", + "/tmp/source.asar", ); - it("preserves filesystem errors raised while consuming a member stream", async () => { - const ioError = Object.assign(new Error("device read failed"), { - code: "EIO", - errno: -5, - syscall: "read", - }); - const source = Readable.from( - (async function* () { - yield Buffer.from("partial"); - throw ioError; - })(), - ); - const output = readValidatedAsarEntry( - source, - undefined, - "nested/member.bin", - "/tmp/source.asar", - ); - - await expect(async () => { - for await (const _chunk of output) { - // Consume through the source failure. - } - }).rejects.toMatchObject({ - name: "ArtifactReaderFailure", - reason: "io", - message: expect.stringContaining( - "Could not read nested/member.bin ASAR at /tmp/source.asar", - ), - cause: { code: "EIO", errno: -5, syscall: "read" }, - }); + await expect(async () => { + for await (const _chunk of output) { + // Consume through the source failure. + } + }).rejects.toMatchObject({ + name: "ArtifactReaderFailure", + reason: "io", + message: expect.stringContaining( + "Could not read nested/member.bin ASAR at /tmp/source.asar", + ), + cause: { code: "EIO", errno: -5, syscall: "read" }, }); }); diff --git a/tests/boundary/filesystem/qwenClientConfiguration.test.ts b/tests/boundary/filesystem/qwenClientConfiguration.test.ts index 0fae29640f..55e8d0ab39 100644 --- a/tests/boundary/filesystem/qwenClientConfiguration.test.ts +++ b/tests/boundary/filesystem/qwenClientConfiguration.test.ts @@ -7,6 +7,7 @@ import { parse as parseJsonc } from "jsonc-parser"; import { readClientRegistrationStatuses } from "../../../src/application/ClientRegistrationStatus.js"; import { configureClientConfiguration } from "../../../src/application/SetupClientConfiguration.js"; import { systemSetupHost } from "../../../src/application/SetupHost.js"; +import { MANAGED_SKILL_FILES } from "../../../src/application/SetupSkill.js"; import { supportedClients } from "../../../src/application/SupportedClients.js"; import { systemUninstallHost } from "../../../src/application/Uninstall.js"; import { PRODUCT_IDENTITY } from "../../../src/identity.js"; @@ -151,6 +152,7 @@ describe("Qwen Code profile setup", () => { }); expect(await setup.installSkill([client.name])).toBe("installed"); const skill = join(home, ".agents", "skills", PRODUCT_IDENTITY.skillName); + await writeFile(join(skill, "user-notes.txt"), "user-authored"); expect(await readFile(join(skill, "SKILL.md"), "utf8")).toContain( "reverse-engineer-anything", ); @@ -173,7 +175,13 @@ describe("Qwen Code profile setup", () => { expect(registered).toEqual(client); expect((await uninstall.removeClient(client)).status).toBe("removed"); expect((await uninstall.removeSkill()).status).toBe("removed"); - await expect(access(skill)).rejects.toMatchObject({ code: "ENOENT" }); + for (const relativePath of MANAGED_SKILL_FILES) + await expect(access(join(skill, relativePath))).rejects.toMatchObject({ + code: "ENOENT", + }); + expect(await readFile(join(skill, "user-notes.txt"), "utf8")).toBe( + "user-authored", + ); if (customHome) await expect(access(join(home, ".qwen"))).rejects.toMatchObject({ code: "ENOENT", diff --git a/tests/boundary/filesystem/safeOutputTree.test.ts b/tests/boundary/filesystem/safeOutputTree.test.ts index 87c0846406..c9b015b6ea 100644 --- a/tests/boundary/filesystem/safeOutputTree.test.ts +++ b/tests/boundary/filesystem/safeOutputTree.test.ts @@ -2,10 +2,14 @@ import { createHash } from "node:crypto"; import { access, chmod, + mkdir, readFile, readdir, readlink, + rename, rm, + symlink, + writeFile, } from "node:fs/promises"; import { join } from "node:path"; import { Readable } from "node:stream"; @@ -17,7 +21,7 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; import { SafeOutputTree } from "../../../src/artifacts/SafeOutputTree.js"; import { ArtifactReaderFailure } from "../../../src/artifacts/ArtifactReader.js"; -describe("safe artifact output tree", () => { +describe("safe artifact output cleanup", () => { it.skipIf(process.platform !== "linux")( "closes the parent descriptor when the output disappears before commit", async () => { @@ -32,12 +36,169 @@ describe("safe artifact output tree", () => { ), ); expect(targets.filter((target) => target === parent)).toEqual([]); + const failure = await tree.rollback().catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(ArtifactReaderFailure); + expect(failure).toMatchObject({ cleanup: { resources: [output] } }); + expect(tree.cleanup).toEqual({ + status: "incomplete", + residualPaths: ["published"], + }); + await rm(parent, { recursive: true, force: true }); + }, + ); +}); + +describe("safe artifact output tree identity", () => { + it.skipIf(process.platform === "win32")( + "refuses a replaced root and retries cleanup when the owned root is restored", + async () => { + const parent = await createTestTempDirectory("rea-safe-output-root-"); + const output = join(parent, "published"); + const displaced = join(parent, "owned-original"); + const outside = join(parent, "outside"); + await mkdir(outside); + const tree = await SafeOutputTree.create(output); + await rename(output, displaced); + await symlink(outside, output, "dir"); + + const writeFailure = await tree + .write("escaped.txt", Readable.from(Buffer.from("outside")), { + sha256: createHash("sha256").update("outside").digest("hex"), + bytes: Buffer.byteLength("outside"), + }) + .catch((cause: unknown) => cause); + expect(writeFailure).toBeInstanceOf(ArtifactReaderFailure); + await expect(access(join(outside, "escaped.txt"))).rejects.toThrow(); + await expect(tree.commit()).rejects.toBeInstanceOf(ArtifactReaderFailure); + const cleanupFailure = await tree + .rollback() + .catch((cause: unknown) => cause); + expect(cleanupFailure).toBeInstanceOf(ArtifactReaderFailure); + expect(cleanupFailure).toMatchObject({ + cleanup: { resources: [output] }, + }); + expect(await readlink(output)).toBe(outside); + + await rm(output); + await rename(displaced, output); expect(await tree.rollback()).toEqual({ status: "complete", residualPaths: [], }); + expect(await readdir(parent)).toEqual(["outside"]); }, ); + + it.skipIf(process.platform === "win32")( + "refuses a replaced nested parent without touching it and retries after restoration", + async () => { + const parent = await createTestTempDirectory("rea-safe-output-parent-"); + const output = join(parent, "published"); + const nested = join(output, "nested"); + const displaced = join(parent, "owned-nested"); + const outside = join(parent, "outside"); + await mkdir(outside); + await writeFile(join(outside, "sentinel.txt"), "keep"); + const tree = await SafeOutputTree.create(output); + const bytes = Buffer.from("owned"); + await tree.write("nested/owned.txt", Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + await rename(nested, displaced); + await symlink(outside, nested, "dir"); + + const writeFailure = await tree + .write("nested/escaped.txt", Readable.from(Buffer.from("outside")), { + sha256: createHash("sha256").update("outside").digest("hex"), + bytes: Buffer.byteLength("outside"), + }) + .catch((cause: unknown) => cause); + expect(writeFailure).toBeInstanceOf(ArtifactReaderFailure); + await expect(access(join(outside, "escaped.txt"))).rejects.toThrow(); + expect(await readFile(join(outside, "sentinel.txt"), "utf8")).toBe( + "keep", + ); + const cleanupFailure = await tree + .rollback() + .catch((cause: unknown) => cause); + expect(cleanupFailure).toBeInstanceOf(ArtifactReaderFailure); + expect(await readlink(nested)).toBe(outside); + expect(await readFile(join(outside, "sentinel.txt"), "utf8")).toBe( + "keep", + ); + + await rm(nested); + await rename(displaced, nested); + expect(await tree.rollback()).toEqual({ + status: "complete", + residualPaths: [], + }); + expect(await readdir(parent)).toEqual(["outside"]); + }, + ); + + it.skipIf(process.platform === "win32")( + "preserves an outside file when a tracked output file is replaced by a symlink", + async () => { + const parent = await createTestTempDirectory("rea-safe-output-file-"); + const output = join(parent, "published"); + const tracked = join(output, "nested", "file.txt"); + const displaced = join(parent, "owned-file.txt"); + const outside = join(parent, "outside.txt"); + await writeFile(outside, "outside bytes"); + const tree = await SafeOutputTree.create(output); + const bytes = Buffer.from("owned bytes"); + await tree.write("nested/file.txt", Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + await rename(tracked, displaced); + await symlink(outside, tracked, "file"); + + const failure = await tree.rollback().catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(ArtifactReaderFailure); + expect(tree.cleanup).toMatchObject({ status: "incomplete" }); + expect(await readlink(tracked)).toBe(outside); + expect(await readFile(outside, "utf8")).toBe("outside bytes"); + + await rm(tracked); + await rename(displaced, tracked); + expect(await tree.rollback()).toEqual({ + status: "complete", + residualPaths: [], + }); + expect(await readFile(outside, "utf8")).toBe("outside bytes"); + }, + ); +}); + +it("retains an untracked regular entry and retries after it is removed", async () => { + const parent = await createTestTempDirectory("rea-safe-output-untracked-"); + const output = join(parent, "published"); + const untracked = join(output, "untracked.txt"); + const tree = await SafeOutputTree.create(output); + const bytes = Buffer.from("owned bytes"); + await tree.write("owned.txt", Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + await writeFile(untracked, "leave untouched"); + + const failure = await tree.rollback().catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(ArtifactReaderFailure); + expect(tree.cleanup).toMatchObject({ status: "incomplete" }); + expect(await readdir(output)).toEqual(["untracked.txt"]); + expect(await readFile(untracked, "utf8")).toBe("leave untouched"); + + await rm(untracked); + expect(await tree.rollback()).toEqual({ + status: "complete", + residualPaths: [], + }); + expect(await readdir(parent)).toEqual([]); +}); +describe("safe artifact output tree", () => { it("removes only its owned tree after digest failure and proves absence", async () => { const parent = await createTestTempDirectory("rea-safe-output-"); const output = join(parent, "published"); @@ -72,6 +233,7 @@ describe("safe artifact output tree", () => { const failure = await tree.rollback().catch((cause: unknown) => cause); expect(failure).toBeInstanceOf(ArtifactReaderFailure); expect(failure).toMatchObject({ + cause: { code: "EACCES" }, cleanup: { resources: [output] }, }); } finally { @@ -81,6 +243,54 @@ describe("safe artifact output tree", () => { }, ); + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "retains cleanup progress when one owned sibling is not writable", + async () => { + const parent = await createTestTempDirectory("rea-safe-output-siblings-"); + const output = join(parent, "published"); + const completed = join(output, "completed"); + const blocked = join(output, "blocked"); + const tree = await SafeOutputTree.create(output); + for (const directory of ["completed", "blocked"]) { + const bytes = Buffer.from(join(output, directory)); + await tree.write(`${directory}/file.txt`, Readable.from(bytes), { + sha256: createHash("sha256").update(bytes).digest("hex"), + bytes: bytes.byteLength, + }); + } + + try { + await chmod(blocked, 0o500); + const failure = await tree.rollback().catch((cause: unknown) => cause); + expect(failure).toBeInstanceOf(ArtifactReaderFailure); + expect(failure).toMatchObject({ + cause: { code: "EACCES" }, + cleanup: { resources: [output] }, + }); + expect(await readdir(output)).toEqual(["blocked"]); + await expect(access(completed)).rejects.toThrow(); + expect(await readFile(join(blocked, "file.txt"), "utf8")).toBe(blocked); + await expect( + tree.write("new/file.txt", Readable.from(Buffer.from("late")), { + sha256: createHash("sha256").update("late").digest("hex"), + bytes: Buffer.byteLength("late"), + }), + ).rejects.toBeInstanceOf(ArtifactReaderFailure); + await expect(tree.commit()).rejects.toBeInstanceOf( + ArtifactReaderFailure, + ); + } finally { + await chmod(blocked, 0o700); + } + + expect(await tree.rollback()).toEqual({ + status: "complete", + residualPaths: [], + }); + expect(await readdir(parent)).toEqual([]); + }, + ); + it("publishes files while building and preserves them after sealing", async () => { const parent = await createTestTempDirectory("rea-safe-output-"); const output = join(parent, "published"); @@ -115,6 +325,13 @@ describe("safe artifact output tree", () => { status: "complete", residualPaths: [], }); + await expect( + tree.write("late.txt", Readable.from(Buffer.from("late")), { + sha256: createHash("sha256").update("late").digest("hex"), + bytes: Buffer.byteLength("late"), + }), + ).rejects.toBeInstanceOf(ArtifactReaderFailure); + await expect(tree.commit()).rejects.toBeInstanceOf(ArtifactReaderFailure); }); it("publishes without POSIX-only directory chmod or fsync on Windows", async () => { diff --git a/tests/boundary/filesystem/setupSkill.test.ts b/tests/boundary/filesystem/setupSkill.test.ts index 40bbcfd825..4da0225c34 100644 --- a/tests/boundary/filesystem/setupSkill.test.ts +++ b/tests/boundary/filesystem/setupSkill.test.ts @@ -187,12 +187,11 @@ describe("canonical skill transaction", () => { expect(installedSkill).toContain( `version: "${PRODUCT_IDENTITY.skillVersion}"`, ); - expect(installedSkill).toContain("call available analysis tools"); - expect(installedSkill).toContain("obtain approval before setup writes"); expect(await readFile(`${nativeGuide}.rea.backup`, "utf8")).toBe( "stale filesystem-write permission grant\n", ); for (const reference of [ + "connection-and-recovery.md", "native-and-artifacts.md", "javascript-applications.md", "android-applications.md", @@ -243,6 +242,7 @@ describe("canonical skill transaction", () => { const paths = [ "SKILL.md", "references/android-applications.md", + "references/connection-and-recovery.md", "references/evidence-workflows.md", "references/javascript-applications.md", "references/native-and-artifacts.md", diff --git a/tests/boundary/firmware/firmwareIntegration.test.ts b/tests/boundary/firmware/firmwareIntegration.test.ts index 3924f37aba..4cce5a73db 100644 --- a/tests/boundary/firmware/firmwareIntegration.test.ts +++ b/tests/boundary/firmware/firmwareIntegration.test.ts @@ -324,7 +324,7 @@ it("cancels a waiting request promptly while keeping later launches behind the a await assertFirmwareCleanup(fixture.launches); }); -it("retains an uncertain workspace and lets the next operation create a new root", async () => { +it("retains an uncertain worker's workspace and blocks later launches", async () => { const fixture = await firmwareFixture("cleanup-failure"); const first = await fixture.service.execute("inspect_firmware_regions", { path: fixture.path, @@ -335,18 +335,22 @@ it("retains an uncertain workspace and lets the next operation create a new root }); const workspace = fixture.launches[0]?.cwd; expect(workspace).toBeDefined(); - if (workspace === undefined) return; + if (workspace === undefined) throw new Error("Expected retained workspace"); await access(workspace); + const originalLaunches = [...fixture.launches]; const next = await fixture.service.execute("inspect_firmware_regions", { path: fixture.path, }); expect(next).toMatchObject({ ok: false, error: { cleanupIncomplete: true } }); - expect(fixture.launches).toHaveLength(2); - const nextWorkspace = fixture.launches[1]?.cwd; - expect(nextWorkspace).toBeDefined(); - expect(nextWorkspace).not.toBe(workspace); + expect(fixture.launches).toEqual(originalLaunches); await access(workspace); - if (nextWorkspace !== undefined) await access(nextWorkspace); + fixture.restoreCleanup(); + const resumed = await fixture.service.execute("inspect_firmware_regions", { + path: fixture.path, + }); + if (!resumed.ok) throw resumed.error; + expect(resumed.value.subject?.local_path).toBe(fixture.path); + await assertFirmwareCleanup(fixture.launches); }); it("accepts another build on the verified release line and reports it", async () => { diff --git a/tests/boundary/firmware/firmwareWorkspaceRecovery.test.ts b/tests/boundary/firmware/firmwareWorkspaceRecovery.test.ts new file mode 100644 index 0000000000..686928a129 --- /dev/null +++ b/tests/boundary/firmware/firmwareWorkspaceRecovery.test.ts @@ -0,0 +1,164 @@ +import { access, chmod, readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { createHash } from "node:crypto"; +import { expect, it } from "vitest"; +import { STDIO_DEFAULT_MAX_BUFFER_SIZE } from "@modelcontextprotocol/server"; + +import { toolContract } from "../../../src/contracts/toolContracts.js"; +import { firmwareResultSchemas } from "../../../src/domain/firmware/firmwareAnalysis.js"; +import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js"; +import { ToolResultDelivery } from "../../../src/server/toolResult.js"; +import { + assertFirmwareCleanup, + firmwareFixture, +} from "../../fixtures/firmware/provider.js"; + +it.skipIf(process.platform !== "linux" || process.getuid?.() === 0)( + "retains completed extractions and retries all workspaces without blocking stopped-worker operations", + async () => { + const fixture = await firmwareFixture("workspace-cleanup-failure"); + const sourceBytes = await readFile(fixture.path); + const sourceSha256 = createHash("sha256").update(sourceBytes).digest("hex"); + const workspaces = new Set(); + try { + for (const output of [ + fixture.output, + join(fixture.root, "second-output"), + ]) { + const extracted = await fixture.service.execute("extract_firmware", { + path: fixture.path, + output_directory: output, + range: { offset: 2, length: 4 }, + }); + const workspace = fixture.launches.at(-1)?.cwd; + if (workspace === undefined) + throw new Error("Expected owned workspace"); + expect(workspaces.has(workspace)).toBe(false); + workspaces.add(workspace); + if (extracted.ok) throw new Error("Expected workspace removal failure"); + expect(extracted.error.cleanupIncomplete).toBe(true); + const partial = extracted.error.partialObservation; + if ( + partial === undefined || + !("kind" in partial) || + partial.kind !== "firmware" + ) + throw new Error("Expected completed firmware observation"); + expect(partial.provenance).toMatchObject({ + subject: { + format: "file", + path: fixture.path, + sha256: sourceSha256, + }, + provider: { id: "unblob", version: "26.6.4" }, + raw_result: { + report: expect.any(Array), + execution: { exit_code: 0 }, + }, + locations: [{ kind: "file-offset-range", start: 2, end: 6 }], + }); + const result = firmwareResultSchemas.extract_firmware.parse( + partial.result, + ); + expect(result.selection).toEqual({ + offset: 2, + length: 4, + sha256: createHash("sha256") + .update(sourceBytes.subarray(2, 6)) + .digest("hex"), + }); + const file = result.files[0]; + if (file === undefined) + throw new Error("Expected published extracted file"); + expect(await readFile(file.path, "utf8")).toBe("firmware=true\n"); + expect(extracted.error.cleanupResources).toContain(workspace); + await access(workspace); + const delivered = new ToolResultDelivery( + STDIO_DEFAULT_MAX_BUFFER_SIZE, + ).toCallToolResult(extracted, toolContract("extract_firmware")); + expect(delivered.isError).toBe(true); + expect(delivered.structuredContent).toMatchObject({ + error: { + details: { + partial_observation: { + kind: "firmware", + result, + provenance: partial.provenance, + }, + }, + }, + }); + + const inspected = await fixture.service.execute( + "inspect_firmware_regions", + { + path: fixture.path, + }, + ); + if (!inspected.ok) throw inspected.error; + expect(inspected.value.subject?.local_path).toBe(fixture.path); + expect( + firmwareResultSchemas.inspect_firmware_regions.parse( + inspected.value.normalized_result, + ).regions[0]?.signature, + ).toBe("gzip"); + } + await expect(fixture.close()).rejects.toMatchObject({ + cleanupResources: expect.arrayContaining([...workspaces]), + }); + for (const workspace of workspaces) await access(workspace); + } finally { + for (const workspace of workspaces) + await chmod(join(workspace, "cleanup-blocked"), 0o700); + await fixture.close(); + } + await assertFirmwareCleanup(fixture.launches); + expect(await readFile(fixture.path)).toEqual(sourceBytes); + for (const output of [fixture.output, join(fixture.root, "second-output")]) + expect(await readFile(join(output, "rootfs/config"), "utf8")).toBe( + "firmware=true\n", + ); + }, +); + +it.skipIf(process.platform !== "linux" || process.getuid?.() === 0)( + "preserves the report failure alongside a workspace removal failure", + async () => { + const fixture = await firmwareFixture( + "workspace-cleanup-failure-malformed", + ); + let workspace: string | undefined; + try { + const failed = await fixture.service.execute("extract_firmware", { + path: fixture.path, + output_directory: fixture.output, + }); + workspace = fixture.launches.at(-1)?.cwd; + if (failed.ok || workspace === undefined) + throw new Error("Expected both report and cleanup failures"); + expect(projectAnalysisError(failed.error)).toMatchObject({ + code: "cleanup_incomplete", + details: { + diagnostics: { + primary_error: { code: "unreadable_output" }, + cleanup_error: { + code: "cleanup_incomplete", + details: { + diagnostics: { reason: expect.stringContaining("EACCES") }, + }, + }, + }, + resources: expect.arrayContaining([workspace]), + }, + }); + await expect(access(fixture.output)).rejects.toMatchObject({ + code: "ENOENT", + }); + } finally { + if (workspace !== undefined) + await chmod(join(workspace, "cleanup-blocked"), 0o700); + await fixture.close(); + } + await assertFirmwareCleanup(fixture.launches); + }, +); diff --git a/tests/boundary/mcp/analysisViewMcp.test.ts b/tests/boundary/mcp/analysisViewMcp.test.ts index 3c15312aa6..0de1f0b026 100644 --- a/tests/boundary/mcp/analysisViewMcp.test.ts +++ b/tests/boundary/mcp/analysisViewMcp.test.ts @@ -19,6 +19,7 @@ import { analysisViewJavaScriptEvidence, analysisViewJavaScriptAnalysisWithSource, analysisViewBindJavaScriptGraphs, + analysisViewLayoutEvidence, analysisViewLayoutFixture, } from "../../fixtures/analysisView.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; @@ -111,6 +112,35 @@ it("advertises exact schemas and projects one layout section from retained Evide ); }); +it("reports a tampered inline Evidence ID at its full MCP source path", async () => { + const { client } = await connect(); + const parent = analysisViewLayoutEvidence(); + const response = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "inline", + evidence: { ...parent, evidence_id: `ev_${"0".repeat(64)}` }, + }, + view: { kind: "summary" }, + }, + }); + + expect(parseMcpToolError(response).error).toMatchObject({ + code: "invalid_request", + category: "invalid_input", + details: { + issues: [ + { + path: ["source", "evidence", "evidence_id"], + reason: "invalid_value", + message: "Evidence semantic identifier does not match its record", + }, + ], + }, + }); +}); + it("delivers a bounded native dossier view over MCP without a Ghidra provider or lost connection", async () => { const { client, session } = await connect(); const original = functionDossierSchema.parse(ghidraFunctionDossier()); diff --git a/tests/boundary/process/darwinProcessRunTokenReader.test.ts b/tests/boundary/process/darwinProcessRunTokenReader.test.ts index 6e438f6c79..6c0025d78c 100644 --- a/tests/boundary/process/darwinProcessRunTokenReader.test.ts +++ b/tests/boundary/process/darwinProcessRunTokenReader.test.ts @@ -36,8 +36,8 @@ const writeBlockingCompiler = async (directory: string) => { const statePath = join(directory, "compile-state.json"); const releasePath = join(directory, "release-compiler"); const source = `#!/usr/bin/env node -const fs = require("node:fs"); -const path = require("node:path"); +const fs = process.getBuiltinModule("node:fs"); +const path = process.getBuiltinModule("node:path"); const statePath = ${JSON.stringify(statePath)}; const releasePath = ${JSON.stringify(releasePath)}; const state = fs.existsSync(statePath) @@ -69,14 +69,14 @@ const writeBlockingIdentityCompiler = async (directory: string) => { const executable = join(directory, "fake-identity-xcrun"); const statePath = join(directory, "identity-helper-state.json"); const helperSource = `#!/usr/bin/env node -const fs = require("node:fs"); +const fs = process.getBuiltinModule("node:fs"); fs.writeFileSync(${JSON.stringify(statePath)}, JSON.stringify({ - root: __dirname, + root: process.getBuiltinModule("node:path").dirname(fs.realpathSync(process.argv[1])), })); setInterval(() => {}, 1000); `; const compilerSource = `#!/usr/bin/env node -const fs = require("node:fs"); +const fs = process.getBuiltinModule("node:fs"); const outputIndex = process.argv.indexOf("-o"); const output = process.argv[outputIndex + 1]; fs.writeFileSync(output, ${JSON.stringify(helperSource)}); @@ -309,14 +309,14 @@ const writeTemporaryObjectCompiler = async (directory: string) => { const executable = join(directory, "fake-temporary-xcrun"); const statePath = join(directory, "temporary-state.json"); const source = `#!/usr/bin/env node -const fs = require("node:fs"); -const os = require("node:os"); -const path = require("node:path"); -const { spawn } = require("node:child_process"); +const fs = process.getBuiltinModule("node:fs"); +const os = process.getBuiltinModule("node:os"); +const path = process.getBuiltinModule("node:path"); +const { spawn } = process.getBuiltinModule("node:child_process"); const temporary = fs.mkdtempSync(path.join(os.tmpdir(), "TemporaryDirectory.")); const object = path.join(temporary, "ProcessRunTokenReader-1.o"); const job = spawn(process.execPath, ["-e", - "setTimeout(() => { const fs = require('node:fs'); fs.mkdirSync(" + JSON.stringify(temporary) + + "setTimeout(() => { const fs = process.getBuiltinModule('node:fs'); fs.mkdirSync(" + JSON.stringify(temporary) + ", { recursive: true }); fs.writeFileSync(" + JSON.stringify(object) + ", 'object'); }, 300)"], { detached: true, stdio: "ignore" }); job.unref(); @@ -567,9 +567,9 @@ it.skipIf(process.platform === "win32")( }, ); -it.skipIf(process.platform === "win32")( - "preserves cancellation through system-host baseline identity inspection", - async () => { +it.skipIf(process.platform === "win32").each(["baseline", "tokens"] as const)( + "preserves cancellation through system-host %s inspection", + async (kind) => { const directory = await mkdtemp( join(tmpdir(), "rea-process-baseline-abort-test-"), ); @@ -579,8 +579,23 @@ it.skipIf(process.platform === "win32")( }); const controller = new AbortController(); let helperRoot: string | undefined; + const inspection = + kind === "baseline" + ? host.captureBaseline?.(controller.signal) + : host.runTokens?.( + [ + { + pid: process.pid, + parentPid: process.ppid, + processGroupId: process.pid, + state: "S", + command: process.execPath, + }, + ], + controller.signal, + ); const baseline = - host.captureBaseline?.(controller.signal).then( + inspection?.then( () => ({ state: "fulfilled" as const }), (cause: unknown) => ({ state: "rejected" as const, cause }), ) ?? Promise.resolve({ state: "missing" as const }); diff --git a/tests/boundary/process/jadxIntegration.test.ts b/tests/boundary/process/jadxIntegration.test.ts index 93ffa475bc..e8c296c670 100644 --- a/tests/boundary/process/jadxIntegration.test.ts +++ b/tests/boundary/process/jadxIntegration.test.ts @@ -91,8 +91,9 @@ it("reuses immutable engine state while keeping each Evidence observation separa await verifyCleanup(launches); }); -it("bounds output per operation across a retained session rather than accumulating earlier captures", async () => { - const { service, apk, launches, provider } = await setup("large-manifest"); +it("bounds output per operation without repeating prerequisite probes for a retained session", async () => { + const { service, apk, launches, provider, javaProbeCount } = + await setup("large-manifest"); const expectedManifest = "x".repeat(1024 * 1024); for (let index = 0; index < 34; index += 1) { const result = await service.execute("inspect_android_package", { @@ -106,12 +107,14 @@ it("bounds output per operation across a retained session rather than accumulati ).toBe(expectedManifest); } expect(launches).toHaveLength(1); + expect(await javaProbeCount()).toBe(1); await provider.close(); await verifyCleanup(launches); }); it("retire sessions when APK bytes, engine bytes or JVM options change", async () => { - const { service, apk, jar, launches, provider, environment } = await setup(); + const { service, apk, jar, launches, provider, environment, javaProbeCount } = + await setup(); const inspect = async () => { const result = await service.execute("inspect_android_package", { path: apk, @@ -133,6 +136,7 @@ it("retire sessions when APK bytes, engine bytes or JVM options change", async ( environment._JAVA_OPTIONS = "-XX:+UseG1GC"; await inspect(); expect(launches).toHaveLength(4); + expect(await javaProbeCount()).toBe(4); expect(launches[3]).toMatchObject({ javaToolOptions: "-Xmx8g", legacyJavaOptions: "-XX:+UseG1GC", diff --git a/tests/boundary/process/processSessionMcp.test.ts b/tests/boundary/process/processSessionMcp.test.ts index 8b6791df9c..7717c7f4e3 100644 --- a/tests/boundary/process/processSessionMcp.test.ts +++ b/tests/boundary/process/processSessionMcp.test.ts @@ -1,8 +1,10 @@ import { fileURLToPath } from "node:url"; +import { access } from "node:fs/promises"; +import { join } from "node:path"; import { Client, InMemoryTransport } from "@modelcontextprotocol/client"; import { Ajv2020 } from "ajv/dist/2020.js"; -import { afterEach, expect } from "vitest"; +import { afterEach, expect, it } from "vitest"; import { z } from "zod"; import { toolContract } from "../../../src/contracts/toolContracts.js"; @@ -13,6 +15,7 @@ import { import { createServer } from "../../../src/server/createServer.js"; import { silentLogger } from "../../../src/logger.js"; import { itWithCaptureCapability } from "./processCaptureCapability.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; const processFixture = fileURLToPath( new URL("../../fixtures/processFidelity.mjs", import.meta.url), @@ -23,6 +26,50 @@ afterEach(async () => { await Promise.all(resources.splice(0).map((resource) => resource.close())); }); +it("rejects oversized initial and resize terminal buffers through MCP before launching", async () => { + const root = await createTestTempDirectory("rea-terminal-admission-"); + const marker = join(root, "launched"); + const session = createTestBinarySession(() => { + throw new Error("Terminal admission must not launch a binary provider"); + }); + const server = createServer( + { kind: "session", session }, + { logger: silentLogger }, + ); + const client = new Client({ name: "terminal-admission", version: "1" }); + resources.push(client, server, session); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + await client.connect(clientTransport); + for (const input of [ + { terminal: { columns: 1_000, rows: 1, scrollback: 1_000 } }, + { events: [{ type: "resize", at_ms: 10, columns: 1_000, rows: 1 }] }, + ] as const) { + const result = await client.callTool({ + name: "capture_process_scenario", + arguments: { + executable: process.execPath, + arguments: [ + "-e", + "require('node:fs').writeFileSync(process.argv[1], 'launched')", + marker, + ], + ...input, + }, + }); + expect(result.isError).toBe(true); + expect(result.content).toContainEqual( + expect.objectContaining({ + type: "text", + text: expect.stringContaining("1000000"), + }), + ); + await expect(access(marker)).rejects.toMatchObject({ code: "ENOENT" }); + } + await client.ping(); +}); + itWithCaptureCapability( "records process residuals linked to capture Evidence", async () => { diff --git a/tests/boundary/process/providerProcess.test.ts b/tests/boundary/process/providerProcess.test.ts index f6143dc5c2..bdc75aea88 100644 --- a/tests/boundary/process/providerProcess.test.ts +++ b/tests/boundary/process/providerProcess.test.ts @@ -202,9 +202,11 @@ describe("provider process output and cleanup primitives", () => { expect(snapshot).toMatchObject({ stdout: { bytes: outputBytes, + observedBytes: outputBytes, }, stderr: { bytes: outputBytes, + observedBytes: outputBytes, }, exitCode: 23, signal: null, diff --git a/tests/boundary/process/providerProcessOutput.test.ts b/tests/boundary/process/providerProcessOutput.test.ts index 888394c99b..c3f4334e02 100644 --- a/tests/boundary/process/providerProcessOutput.test.ts +++ b/tests/boundary/process/providerProcessOutput.test.ts @@ -29,6 +29,12 @@ it("bounds capture before diagnostic callbacks while draining both output stream expect( diagnostics.some((event) => event.type === "output" && event.truncated), ).toBe(true); + const snapshot = supervisor.snapshot(); + expect(snapshot.stdout.observedBytes).toBe(70_000); + expect(snapshot.stderr.observedBytes).toBe(70_000); + expect(snapshot.stdout.bytes + snapshot.stderr.bytes).toBeLessThanOrEqual( + 1024, + ); for (const stream of ["stdout", "stderr"]) expect( diagnostics diff --git a/tests/boundary/providers/ghidra/ghidraClient.test.ts b/tests/boundary/providers/ghidra/ghidraClient.test.ts index ac2e5cbc16..bcb4de53f0 100644 --- a/tests/boundary/providers/ghidra/ghidraClient.test.ts +++ b/tests/boundary/providers/ghidra/ghidraClient.test.ts @@ -11,6 +11,7 @@ import type { Logger } from "pino"; import { ok } from "../../../../src/domain/result.js"; import { GhidraClient } from "../../../../src/ghidra/GhidraClient.js"; +import { GHIDRA_PROCESS_DIAGNOSTIC_BYTES } from "../../../../src/ghidra/GhidraDiagnostics.js"; import { GhidraHeadlessLauncher } from "../../../../src/ghidra/GhidraLauncher.js"; import type { GhidraDiagnostic } from "../../../../src/ghidra/GhidraClientTypes.js"; import type { @@ -49,7 +50,8 @@ type FixtureMode = | "hang_tools" | "exit_tools" | "silent" - | "exit"; + | "exit" + | "diagnostic_output"; class FixtureLauncher implements GhidraLauncher { #notifyStarted: ((child: ChildProcess) => void) | undefined; @@ -458,6 +460,53 @@ describe("GhidraClient cleanup and diagnostics", () => { }); }); + it("bounds long-lived launcher diagnostics without interrupting the session", async () => { + const launcher = new FixtureLauncher("diagnostic_output"); + const client = clientFor(launcher); + await expect(client.start()).resolves.toMatchObject({ ok: true }); + await expect( + client.callTool("list_strings", { document: null }), + ).resolves.toMatchObject({ ok: true }); + + await vi.waitFor(() => { + expect(client.diagnostics()).toMatchObject({ + diagnostic_truncated: true, + }); + }); + const diagnostics = client.diagnostics(); + const stdout = diagnostics.stdout as { + bytes: number; + retained_bytes: number; + text: string; + }; + const stderr = diagnostics.stderr as { + bytes: number; + retained_bytes: number; + text: string; + }; + expect(stdout.bytes + stderr.bytes).toBeGreaterThan( + GHIDRA_PROCESS_DIAGNOSTIC_BYTES, + ); + expect(stdout.retained_bytes + stderr.retained_bytes).toBeLessThanOrEqual( + GHIDRA_PROCESS_DIAGNOSTIC_BYTES, + ); + expect(stdout.bytes).toBeGreaterThanOrEqual(stdout.retained_bytes); + expect(stderr.bytes).toBeGreaterThanOrEqual(stderr.retained_bytes); + expect(JSON.stringify(diagnostics)).not.toContain(launcher.tokens[0] ?? ""); + expect(`${stdout.text}${stderr.text}`).toContain("[REDACTED]"); + expect( + stderr.text.endsWith( + "source=/tmp/local-evidence.bin?cursor=keep&[REDACTED]", + ), + ).toBe(true); + + await client.close(); + expect(exited(launcher.processes[0])).toBe(true); + await expect(access(launcher.runtimeRoots[0] ?? "")).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it("retains actionable diagnostics without retaining its token", async () => { const launcher = new FixtureLauncher("exit"); const events: GhidraDiagnostic[] = []; diff --git a/tests/boundary/providers/hopper/hopperClient.diagnostics.test.ts b/tests/boundary/providers/hopper/hopperClient.diagnostics.test.ts new file mode 100644 index 0000000000..fd4982d9e4 --- /dev/null +++ b/tests/boundary/providers/hopper/hopperClient.diagnostics.test.ts @@ -0,0 +1,88 @@ +import { access } from "node:fs/promises"; + +import { expect, it, onTestFinished, vi } from "vitest"; + +import { HopperTimeoutError } from "../../../../src/domain/hopperErrors.js"; +import { + HopperClient, + HOPPER_PROCESS_DIAGNOSTIC_BYTES, +} from "../../../../src/hopper/HopperClient.js"; +import { HopperFixtureLauncher } from "./hopperClient.fixture.js"; + +it("bounds long-lived launcher output while keeping the bridge usable", async () => { + let observedStderrBytes = 0; + const diagnosticLauncher = new HopperFixtureLauncher(undefined, "noisy"); + const client = new HopperClient({ + launcher: diagnosticLauncher, + startupTimeoutMs: 10_000, + onDiagnostic(event) { + if (event.type === "launcher-stderr") observedStderrBytes += event.bytes; + }, + }); + onTestFinished(async () => { + await client.close(); + }); + + await expect(client.start()).resolves.toMatchObject({ ok: true }); + const session = await client.callTool("echo", { value: "still usable" }); + expect(session).toEqual({ ok: true, value: { value: "still usable" } }); + await vi.waitFor(() => { + expect(observedStderrBytes).toBeGreaterThan( + HOPPER_PROCESS_DIAGNOSTIC_BYTES, + ); + }); + + await client.close(); + const directory = diagnosticLauncher.directories[0] ?? ""; + await expect(access(directory)).rejects.toMatchObject({ code: "ENOENT" }); + expect(diagnosticLauncher.processes[0]?.exitCode).not.toBeNull(); +}); + +it("reports observed and retained output when capture cuts a token", async () => { + const diagnosticLauncher = new HopperFixtureLauncher( + undefined, + "noisy_no_socket", + ); + const client = new HopperClient({ + launcher: diagnosticLauncher, + startupTimeoutMs: 1_000, + }); + onTestFinished(async () => { + await client.close(); + }); + + const started = await client.start(); + if (started.ok || !(started.error instanceof HopperTimeoutError)) + throw new Error("Expected the no-socket fixture to time out"); + const outcome = started.error.launcherOutcome; + expect(outcome).toMatchObject({ diagnostic_truncated: true }); + if (outcome === undefined) throw new Error("Launcher outcome was omitted"); + expect(outcome.stdout.bytes + outcome.stderr.bytes).toBeGreaterThan( + HOPPER_PROCESS_DIAGNOSTIC_BYTES, + ); + expect( + outcome.stdout.retained_bytes + outcome.stderr.retained_bytes, + ).toBeLessThanOrEqual(HOPPER_PROCESS_DIAGNOSTIC_BYTES); + expect(outcome.stdout.bytes).toBeGreaterThanOrEqual( + outcome.stdout.retained_bytes, + ); + expect(outcome.stderr.bytes).toBeGreaterThanOrEqual( + outcome.stderr.retained_bytes, + ); + expect(outcome.stderr.retained_bytes).toBe(HOPPER_PROCESS_DIAGNOSTIC_BYTES); + const token = diagnosticLauncher.tokens[0] ?? ""; + expect(JSON.stringify(outcome)).not.toContain(token); + expect(`${outcome.stdout.text}${outcome.stderr.text}`).toContain( + "[redacted transport credential]", + ); + expect( + outcome.stderr.text.endsWith( + "source=/tmp/local-evidence.bin?cursor=keep&[redacted transport credential]", + ), + ).toBe(true); + + await client.close(); + const directory = diagnosticLauncher.directories[0] ?? ""; + await expect(access(directory)).rejects.toMatchObject({ code: "ENOENT" }); + expect(diagnosticLauncher.processes[0]?.exitCode).not.toBeNull(); +}); diff --git a/tests/boundary/providers/hopper/hopperClient.fixture.ts b/tests/boundary/providers/hopper/hopperClient.fixture.ts index 55b79083e1..7bb4296063 100644 --- a/tests/boundary/providers/hopper/hopperClient.fixture.ts +++ b/tests/boundary/providers/hopper/hopperClient.fixture.ts @@ -28,15 +28,20 @@ export class HopperFixtureLauncher implements BridgeLauncher { readonly socketPaths: string[] = []; readonly directories: string[] = []; readonly runIds: string[] = []; + readonly tokens: string[] = []; readonly processes: ChildProcess[] = []; readonly requests: FixtureRequest[] = []; - constructor(readonly tokenOverride?: string) {} + constructor( + readonly tokenOverride?: string, + readonly diagnosticMode: "quiet" | "noisy" | "noisy_no_socket" = "quiet", + ) {} launch(session: BridgeSession) { this.socketPaths.push(session.socketPath); this.directories.push(session.directory); this.runIds.push(session.runId); + this.tokens.push(session.token); const child = spawn( process.execPath, [ @@ -44,8 +49,10 @@ export class HopperFixtureLauncher implements BridgeLauncher { session.socketPath, this.tokenOverride ?? session.token, session.runId, + "acknowledge", + this.diagnosticMode, ], - { stdio: ["ignore", "ignore", "pipe", "ipc"] }, + { stdio: ["ignore", "pipe", "pipe", "ipc"] }, ); child.on("message", (value: unknown) => { const request = requestSchema.safeParse(value); diff --git a/tests/boundary/providers/native/artifactProvider.machOSlices.test.ts b/tests/boundary/providers/native/artifactProvider.machOSlices.test.ts index 2dc315f484..f46453cc25 100644 --- a/tests/boundary/providers/native/artifactProvider.machOSlices.test.ts +++ b/tests/boundary/providers/native/artifactProvider.machOSlices.test.ts @@ -1,7 +1,7 @@ import { writeFile } from "node:fs/promises"; import { join } from "node:path"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, onTestFinished } from "vitest"; import { createTestTempDirectory } from "../../../fixtures/temporaryDirectory.js"; @@ -40,6 +40,7 @@ describe("artifact Mach-O slices", () => { ), }; const reader = new MachOSliceArtifactReader(binary, {}, runner); + onTestFinished(() => reader.close()); const enumerate = async (): Promise => { for await (const _entry of reader.entries()) { // Enumeration must reject out-of-bounds lipo metadata before yielding. @@ -78,6 +79,7 @@ describe("artifact Mach-O slices", () => { ), }; const reader = new MachOSliceArtifactReader(binary, {}, runner); + onTestFinished(() => reader.close()); const entries = []; for await (const entry of reader.entries()) entries.push(entry); expect(entries).toHaveLength(2); @@ -105,7 +107,9 @@ describe("artifact Mach-O slices", () => { Reflect.set(provenance[0], "tool", "forged"); expect(reader.provenance()[0]?.tool).toBe("lipo"); }); +}); +describe("artifact Mach-O slice integrity", () => { it.each([ ["CPU type", "cputype 16777223"], ["CPU subtype", "cpusubtype 2"], @@ -139,6 +143,7 @@ describe("artifact Mach-O slices", () => { {}, lipoRunner(`${details.join("\n")}\n`), ); + onTestFinished(() => reader.close()); await expect(async () => { for await (const _entry of reader.entries()) { // The structural comparison runs before the reader yields a slice. @@ -161,6 +166,7 @@ describe("artifact Mach-O slices", () => { `architecture arm64\n cputype 16777228\n cpusubtype 0\n offset 4096\n size ${thin.length}\n align 2^12 (4096)\n`, ), ); + onTestFinished(() => reader.close()); await expect(async () => { for await (const _entry of reader.entries()) { // Malformed structural facts must fail before any slice is yielded. @@ -215,6 +221,7 @@ describe("artifact Mach-O arm64e variant slices", () => { `architecture x86_64\n cputype CPU_TYPE_X86_64\n cpusubtype CPU_SUBTYPE_X86_64_ALL\n offset 4096\n size ${String(x86.length)}\n align 2^12 (4096)\n${lipo(arm.length)}`, ), ); + onTestFinished(() => reader.close()); const entries = []; for await (const entry of reader.entries()) entries.push(entry); return entries.map(({ path, byteOffset }) => [path, byteOffset]); diff --git a/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts b/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts index 7355ba0759..25b9e07303 100644 --- a/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts +++ b/tests/boundary/providers/native/nativeSignatureTargetBinding.test.ts @@ -9,6 +9,7 @@ import { rename, rm, symlink, + utimes, writeFile, } from "node:fs/promises"; import { dirname, join } from "node:path"; @@ -102,8 +103,13 @@ describe("native signature target identity", () => { const runner = new Captures(async (args) => { if (!args.includes("--entitlements")) return; if (change === "remove") await rm(path); - else if (change === "rewrite") await writeFile(path, CONTENT); - else { + else if (change === "rewrite") { + const before = await lstat(path); + await writeFile(path, CONTENT); + // Identical writes may share a filesystem clock tick. Make the + // metadata change observed by the target guard explicit. + await utimes(path, before.atimeMs / 1000, before.mtimeMs / 1000 + 1); + } else { const replacement = `${path}.replacement`; await writeFile(replacement, CONTENT); await rename(replacement, path); diff --git a/tests/composition/analysis-sessions/managedApplicationGraph.test.ts b/tests/composition/analysis-sessions/managedApplicationGraph.test.ts index e96efb442a..915545a69e 100644 --- a/tests/composition/analysis-sessions/managedApplicationGraph.test.ts +++ b/tests/composition/analysis-sessions/managedApplicationGraph.test.ts @@ -8,6 +8,10 @@ import { MANAGED_STATIC_PROVIDER } from "../../../src/application/InvestigationP import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { parseJavaScriptApplicationGraph } from "../../../src/domain/javascript/javascriptApplicationGraph.js"; import { managedApplicationGraphResultSchema } from "../../../src/domain/managed/managedApplicationGraph.js"; +import { + managedMemberInspectionSchema, + managedNativeBoundaryInspectionSchema, +} from "../../../src/domain/managed/managedArtifact.js"; import { inspectManagedArtifactBytes } from "../../../src/dotnet/ManagedArtifactInspector.js"; import { inspectManagedMembersBytes } from "../../../src/dotnet/ManagedMemberInspector.js"; import { inspectManagedNativeBoundariesBytes } from "../../../src/dotnet/ManagedNativeBoundaryInspector.js"; @@ -299,6 +303,282 @@ describe("managed application graph identity facts", () => { }); }); +it("normalizes MVID casing for node identity while preserving observed values", () => { + const { memberEvidence, boundaryEvidence } = createManagedInteropEvidence(); + const members = managedMemberInspectionSchema.parse( + parseEvidence(memberEvidence).normalized_result, + ); + if (members.module === null) + throw new Error("Expected fixture module identity"); + const uppercaseMvid = members.module.mvid?.toUpperCase(); + if (uppercaseMvid === undefined || uppercaseMvid === null) + throw new Error("Expected fixture module MVID"); + const uppercaseMembers = createEvidence(undefined, MANAGED_STATIC_PROVIDER, { + operation: "inspect_managed_members", + parameters: {}, + result: { + ...members, + module: { + ...members.module, + mvid: uppercaseMvid, + enc_id: null, + enc_base_id: null, + }, + identity_scope: { + ...members.identity_scope, + requires_mvid: uppercaseMvid, + }, + }, + rawResult: null, + limitations: members.limitations, + }); + const baseline = projectManagedApplicationGraphEvidence({ + managed_members: memberEvidence, + managed_native_boundaries: boundaryEvidence, + }); + const uppercase = projectManagedApplicationGraphEvidence({ + managed_members: uppercaseMembers, + managed_native_boundaries: boundaryEvidence, + }); + if (!baseline.ok) throw baseline.error; + if (!uppercase.ok) throw uppercase.error; + const baselineResult = managedApplicationGraphResultSchema.parse( + parseEvidence(baseline.value).normalized_result, + ); + const uppercaseResult = managedApplicationGraphResultSchema.parse( + parseEvidence(uppercase.value).normalized_result, + ); + const baselineModule = parseJavaScriptApplicationGraph( + baselineResult.graph, + ).nodes.find(({ kind }) => kind === "managed-module"); + const uppercaseGraph = parseJavaScriptApplicationGraph(uppercaseResult.graph); + const uppercaseModule = uppercaseGraph.nodes.find( + ({ kind }) => kind === "managed-module", + ); + expect(uppercaseGraph.edges).toContainEqual( + expect.objectContaining({ + relation: "imports", + source_node_id: uppercaseGraph.nodes.find( + ({ kind }) => kind === "managed-method", + )?.node_id, + }), + ); + expect(uppercaseModule?.node_id).toBe(baselineModule?.node_id); + expect(uppercaseModule?.observations[0]?.properties.mvid).toBe(uppercaseMvid); +}); + +describe("managed application graph module identity consistency", () => { + it("rejects contradictory known module MVIDs across same-artifact inputs", () => { + const { artifactEvidence, memberEvidence, boundaryEvidence } = + createManagedInteropEvidence(); + const boundaries = managedNativeBoundaryInspectionSchema.parse( + parseEvidence(boundaryEvidence).normalized_result, + ); + const conflictingMvid = "ffffffff-ffff-ffff-ffff-ffffffffffff"; + const conflictingBoundaries = createEvidence( + undefined, + MANAGED_STATIC_PROVIDER, + { + operation: "inspect_managed_native_boundaries", + parameters: {}, + result: { + ...boundaries, + module: + boundaries.module === null + ? null + : { ...boundaries.module, mvid: conflictingMvid }, + identity_scope: { + ...boundaries.identity_scope, + requires_mvid: conflictingMvid, + }, + }, + rawResult: null, + limitations: boundaries.limitations, + }, + ); + + const projected = projectManagedApplicationGraphEvidence({ + managed_artifact: artifactEvidence, + managed_members: memberEvidence, + managed_native_boundaries: conflictingBoundaries, + }); + expect(projected.ok).toBe(false); + if (projected.ok) throw new Error("conflicting MVIDs were accepted"); + expect(JSON.stringify(projected.error)).toContain( + "conflicting known module MVIDs or token scopes", + ); + }); + + it("rejects contradictory known module declaration fields", () => { + const { artifactEvidence, memberEvidence } = createManagedInteropEvidence(); + const members = managedMemberInspectionSchema.parse( + parseEvidence(memberEvidence).normalized_result, + ); + if (members.module === null) + throw new Error("Expected fixture module identity"); + const conflictingMembers = createEvidence( + undefined, + MANAGED_STATIC_PROVIDER, + { + operation: "inspect_managed_members", + parameters: {}, + result: { + ...members, + module: { + ...members.module, + generation: members.module.generation + 1, + }, + }, + rawResult: null, + limitations: members.limitations, + }, + ); + + const projected = projectManagedApplicationGraphEvidence({ + managed_artifact: artifactEvidence, + managed_members: conflictingMembers, + }); + expect(projected.ok).toBe(false); + if (projected.ok) + throw new Error("conflicting module declaration was accepted"); + expect(JSON.stringify(projected.error)).toContain( + "conflicting module generation values", + ); + }); +}); + +describe("managed application graph token scope", () => { + it("checks declared token artifact and MVID scopes without a subject digest", () => { + const { memberEvidence } = createManagedInteropEvidence(); + const members = managedMemberInspectionSchema.parse( + parseEvidence(memberEvidence).normalized_result, + ); + const wrongMvidValue = "ffffffff-ffff-ffff-ffff-ffffffffffff"; + const wrongMvidEvidence = createEvidence( + undefined, + MANAGED_STATIC_PROVIDER, + { + operation: "inspect_managed_members", + parameters: {}, + result: { + ...members, + identity_scope: { + ...members.identity_scope, + requires_mvid: wrongMvidValue, + }, + }, + rawResult: null, + limitations: members.limitations, + }, + ); + const wrongArtifactEvidence = createEvidence( + undefined, + MANAGED_STATIC_PROVIDER, + { + operation: "inspect_managed_members", + parameters: {}, + result: { + ...members, + identity_scope: { + ...members.identity_scope, + requires_artifact_sha256: "b".repeat(64), + }, + }, + rawResult: null, + limitations: members.limitations, + }, + ); + + const wrongMvidResult = projectManagedApplicationGraphEvidence({ + managed_members: wrongMvidEvidence, + }); + expect(wrongMvidResult.ok).toBe(false); + if (wrongMvidResult.ok) + throw new Error("mismatched token MVID was accepted"); + expect(JSON.stringify(wrongMvidResult.error)).toContain("token scope MVID"); + + const wrongArtifact = projectManagedApplicationGraphEvidence({ + managed_members: wrongArtifactEvidence, + }); + expect(wrongArtifact.ok).toBe(false); + if (wrongArtifact.ok) + throw new Error("mismatched token artifact digest was accepted"); + expect(JSON.stringify(wrongArtifact.error)).toContain( + "token scope SHA-256", + ); + }); +}); + +describe("unknown managed application graph token scope", () => { + it("keeps absent MVID scope unknown and marks cross-source token joins partial", () => { + const { memberEvidence, boundaryEvidence } = createManagedInteropEvidence(); + const members = managedMemberInspectionSchema.parse( + parseEvidence(memberEvidence).normalized_result, + ); + const boundaries = managedNativeBoundaryInspectionSchema.parse( + parseEvidence(boundaryEvidence).normalized_result, + ); + const unknownMembers = createEvidence(undefined, MANAGED_STATIC_PROVIDER, { + operation: "inspect_managed_members", + parameters: {}, + result: { + ...members, + module: + members.module === null + ? null + : { + ...members.module, + mvid: null, + enc_id: null, + enc_base_id: null, + }, + identity_scope: { ...members.identity_scope, requires_mvid: null }, + }, + rawResult: null, + limitations: members.limitations, + }); + + const projected = projectManagedApplicationGraphEvidence({ + managed_members: unknownMembers, + managed_native_boundaries: boundaryEvidence, + }); + if (!projected.ok) throw projected.error; + const result = managedApplicationGraphResultSchema.parse( + parseEvidence(projected.value).normalized_result, + ); + const graph = parseJavaScriptApplicationGraph(result.graph); + expect(result.summary.modules).toBe(1); + expect(result.coverage.status).toBe("partial"); + expect(result.limitations).toContain( + "Managed boundary tokens were not linked to member nodes because matching module MVID scope was not established.", + ); + const moduleObservation = graph.nodes.find( + ({ kind }) => kind === "managed-module", + )?.observations[0]; + expect(moduleObservation?.properties.mvid).toBe(boundaries.module?.mvid); + expect(moduleObservation?.evidence.extractor.operation).toBe( + "inspect_managed_native_boundaries", + ); + const method = graph.nodes.find( + ({ kind, observations }) => + kind === "managed-method" && + observations[0]?.label === "Fixture.Program.Main", + ); + expect(method).toBeDefined(); + const importEdge = graph.edges.find( + ({ relation, target_node_id }) => + relation === "imports" && + graph.nodes.find(({ node_id }) => node_id === target_node_id)?.kind === + "managed-pinvoke-import", + ); + expect(importEdge).toBeDefined(); + expect( + graph.nodes.find(({ node_id }) => node_id === importEdge?.source_node_id) + ?.kind, + ).toBe("artifact"); + }); +}); + describe("managed application graph coverage", () => { it("preserves partial parser coverage in graph and per-fact coverage", () => { const bytes = buildManagedPeFixture(); diff --git a/tests/composition/runtime-observation/javascriptRuntimeReconciliation.test.ts b/tests/composition/runtime-observation/javascriptRuntimeReconciliation.test.ts index a6809156eb..9e90a1da4d 100644 --- a/tests/composition/runtime-observation/javascriptRuntimeReconciliation.test.ts +++ b/tests/composition/runtime-observation/javascriptRuntimeReconciliation.test.ts @@ -186,6 +186,40 @@ it("matches renderer, frame, script bytes, and worker without claiming execution ); }); +it("keeps static load states unknown when one scoped capture omits scripts", async () => { + const fixture = await applicationFixture(); + const staticEvidence = await analyzeFixture(fixture); + const result = reconcileInput({ + static_layers: [{ role: "application", analysis: staticEvidence }], + runtime_observations: [ + electronRuntimeEvidence(fixture, SOURCE, { + includeWorker: false, + targetId: "target-complete", + }), + electronRuntimeEvidence(fixture, SOURCE, { + includeWorker: false, + scriptsUnavailable: true, + targetId: "target-incomplete", + }), + ], + }); + + expect( + result.runtime_captures.map( + ({ scripts_complete_within_scope }) => scripts_complete_within_scope, + ), + ).toEqual(expect.arrayContaining([false, true])); + expect(result.summary.static_not_observed).toBe(0); + expect(result.static_load_states).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + status: "unknown", + reason: "static-or-runtime-coverage-incomplete", + }), + ]), + ); +}); + it("reports a captured digest disagreement instead of accepting a path match", async () => { const fixture = await applicationFixture(); const staticEvidence = await analyzeFixture(fixture); @@ -316,7 +350,7 @@ it("imports an operator-provided cache layer through an explicit file mapping", const application = await applicationFixture(); const cache = await createTestTempDirectory("rea-runtime-cache-static-"); const runtimeCache = await createTestTempDirectory("rea-runtime-cache-live-"); - const cacheSource = "export const cachedFeature = 'fixture';\n"; + const cacheSource = SOURCE; await mkdir(join(cache, "mapped")); await Promise.all([ writeFile(join(cache, "mapped", "chunk.js"), cacheSource), @@ -373,6 +407,16 @@ it("imports an operator-provided cache layer through an explicit file mapping", status: "unknown", reason: "layer-outside-runtime-scope", }); + const sharedFile = result.graph.nodes.find( + ({ observations }) => + observations.some(({ properties }) => properties.path === "app.js") && + observations.some( + ({ properties }) => properties.path === "mapped/chunk.js", + ), + ); + expect( + sharedFile?.observations.map(({ properties }) => properties.path), + ).toEqual(expect.arrayContaining(["app.js", "mapped/chunk.js"])); }); const applicationFixture = async (): Promise => { @@ -410,6 +454,7 @@ const electronRuntimeEvidence = ( readonly targetId?: string; readonly sourceIncluded?: boolean; readonly workersUnavailable?: boolean; + readonly scriptsUnavailable?: boolean; } = {}, ) => { const scriptFile = options.scriptFile ?? "app.js"; @@ -445,15 +490,7 @@ const electronRuntimeEvidence = ( ended_at: "2026-07-15T00:00:00.100Z", observation_ms: 100, }, - completeness: options.workersUnavailable - ? { - ...completeCapture(), - status: "attach_limited" as const, - conditions: ["attach_limited" as const], - attach_limited_sections: ["workers" as const], - unavailable_sections: ["workers" as const], - } - : completeCapture(), + completeness: captureCompleteness(options), frames: [ { frame_id: "frame-main", @@ -522,3 +559,21 @@ const completeCapture = () => ({ total: 0, }, }); + +const captureCompleteness = (options: { + readonly workersUnavailable?: boolean; + readonly scriptsUnavailable?: boolean; +}) => { + const unavailableSections = [ + ...(options.scriptsUnavailable ? (["scripts"] as const) : []), + ...(options.workersUnavailable ? (["workers"] as const) : []), + ]; + if (unavailableSections.length === 0) return completeCapture(); + return { + ...completeCapture(), + status: "attach_limited" as const, + conditions: ["attach_limited" as const], + attach_limited_sections: unavailableSections, + unavailable_sections: unavailableSections, + }; +}; diff --git a/tests/evaluation/model-evals/agentEvaluationReport.test.ts b/tests/evaluation/model-evals/agentEvaluationReport.test.ts index 1913da97d2..562fd591ea 100644 --- a/tests/evaluation/model-evals/agentEvaluationReport.test.ts +++ b/tests/evaluation/model-evals/agentEvaluationReport.test.ts @@ -77,6 +77,29 @@ async function runnerFailurePredicate(): Promise<(item: unknown) => unknown> { } describe("scoped factual report", () => { + it("requires a validated target clarification and zero REA calls for the negative route", () => { + const clarification = { + ...routingSuccess, + expectedFirstTool: null, + naturalUse: false, + factualCorrectness: "not_assessed" as const, + answerHeuristicsMet: false, + epistemicCuePresent: false, + targetClarificationPassed: true, + }; + expect(agentEvaluationPassed(clarification)).toBe(true); + for (const change of [ + { naturalUse: true }, + { correctFirstTool: false }, + { targetClarificationPassed: false }, + { exitCode: 1 }, + { inputTokens: 0 }, + ]) + expect(agentEvaluationPassed({ ...clarification, ...change })).toBe( + false, + ); + }); + it("accepts correct structured facts without superficial prose cues", () => { expect( agentEvaluationPassed({ diff --git a/tests/evaluation/model-evals/codexAgentEval.test.ts b/tests/evaluation/model-evals/codexAgentEval.test.ts index 61aaa34fbb..f802d68d29 100644 --- a/tests/evaluation/model-evals/codexAgentEval.test.ts +++ b/tests/evaluation/model-evals/codexAgentEval.test.ts @@ -101,6 +101,35 @@ describe("Codex agent release evaluation", () => { ).toBe(1); }); + it("distinguishes session verification after a successful close from an unchanged retry", () => { + const call = (id: string, tool: string, status = "completed") => ({ + type: "item.completed", + item: { + id, + type: "mcp_tool_call", + server: "rea", + tool, + arguments: {}, + status, + }, + }); + for (const [status, repeats] of [ + ["completed", 0], + ["failed", 1], + ] as const) { + expect( + evaluateCodexEvents( + [ + call("before", "binary_session"), + call("close", "close_binary", status), + call("after", "binary_session"), + ], + "binary_session", + ).repeatedCallCount, + ).toBe(repeats); + } + }); + it("counts identical arguments with reversed Unicode key insertion order", () => { const call = (id: string, arguments_: Record) => ({ type: "item.completed", @@ -125,6 +154,119 @@ describe("Codex agent release evaluation", () => { }); }); +describe("Codex agent routing and error evaluation", () => { + it("preserves an unrecognized structured failure without inventing its code", () => { + const metrics = evaluateCodexEvents( + [ + { + type: "item.completed", + item: { + type: "mcp_tool_call", + server: "rea", + tool: "close_binary", + arguments: {}, + result: { structured_content: { error: "unrecognized failure" } }, + }, + }, + ], + "close_binary", + { requiredToolSubsequence: ["close_binary"] }, + ); + expect(metrics.reaCalls[0]).toMatchObject({ error: true, errorCode: null }); + expect(metrics.requiredToolSubsequenceMet).toBe(false); + }); + + it("requires no REA call when target selection is missing", () => { + const events = [ + { + type: "item.completed", + item: { type: "agent_message", text: "Which app should I inspect?" }, + }, + ]; + expect(evaluateCodexEvents(events, null)).toMatchObject({ + naturalUse: false, + correctFirstTool: true, + firstTool: null, + }); + expect( + evaluateCodexEvents( + [ + ...events, + { + type: "item.completed", + item: { + type: "mcp_tool_call", + server: "rea", + tool: "analyze_javascript_application", + arguments: { input_path: "/nearby/test-fixture" }, + }, + }, + ], + null, + ).correctFirstTool, + ).toBe(false); + }); + + it("counts actual text-only REA errors and SDK-rejected schema inputs once per call", () => { + const source = { + kind: "retained-evidence", + evidence_id: `ev_${"a".repeat(64)}`, + }; + const failed = (id: string, arguments_: unknown, text: string) => ({ + type: "item.completed", + item: { + id, + type: "mcp_tool_call", + server: "rea", + tool: "inspect_analysis_view", + arguments: arguments_, + status: "failed", + error: null, + result: { content: [{ type: "text", text }], structured_content: null }, + }, + }); + const metrics = evaluateCodexEvents( + [ + failed( + "sdk", + { source, view: { kind: "module", path: "main.js" } }, + "Input validation error: Invalid arguments for tool inspect_analysis_view: view.kind: Invalid discriminator value.", + ), + failed( + "ambiguous", + { + source, + view: { + kind: "item", + collection: "modules", + selector: { path: "main.js" }, + }, + }, + JSON.stringify({ + error: { + code: "invalid_request", + message: + "Multiple modules have this path; select one by node_id.", + }, + }), + ), + failed( + "incomplete-preview", + { source, view: { kind: "summary" } }, + '{"error":{"code":"invalid_request"', + ), + ], + "inspect_analysis_view", + ); + expect(metrics.inputValidationFailureCount).toBe(2); + expect(metrics.reaCalls.map(({ errorCode }) => errorCode)).toEqual([ + null, + "invalid_request", + null, + ]); + }); +}); + describe("Codex agent completion evaluation", () => { it("does not present matching answer terms as factual correctness or honesty", () => { const evidenceId = `ev_${"a".repeat(64)}`; diff --git a/tests/fixtures/android/jadx.ts b/tests/fixtures/android/jadx.ts index 8e8ac48847..69dc86be52 100644 --- a/tests/fixtures/android/jadx.ts +++ b/tests/fixtures/android/jadx.ts @@ -1,4 +1,11 @@ -import { access, chmod, mkdir, rm, writeFile } from "node:fs/promises"; +import { + access, + chmod, + mkdir, + readFile, + rm, + writeFile, +} from "node:fs/promises"; import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { expect, onTestFinished } from "vitest"; @@ -27,10 +34,11 @@ export const createJadxProtocolFixture = async ( const javaHome = join(root, "jdk"); const javaBin = join(javaHome, "bin"); const java = join(javaBin, "java"); + const javaProbeLog = join(root, "java-probes.log"); await mkdir(javaBin, { recursive: true }); await writeFile( java, - `#!/usr/bin/env node\nif (process.argv[2] === "--list-modules") console.log("java.base@21.0.0\\njdk.compiler@21.0.0");\n`, + `#!/usr/bin/env node\nif (process.argv[2] === "--list-modules") { require("node:fs").appendFileSync(${JSON.stringify(javaProbeLog)}, "probe\\n"); console.log("java.base@21.0.0\\njdk.compiler@21.0.0"); }\n`, ); await chmod(java, 0o755); await writeOrderedZip(apk, ["AndroidManifest.xml", "classes.dex"]); @@ -115,6 +123,10 @@ export const createJadxProtocolFixture = async ( jar, environment, launches, + javaProbeCount: async () => { + const contents = await readFile(javaProbeLog, "utf8"); + return contents.split("\n").filter(Boolean).length; + }, }; }; diff --git a/tests/fixtures/fakeCdpBrowser.ts b/tests/fixtures/fakeCdpBrowser.ts index c085b18062..5c99f1fb0f 100644 --- a/tests/fixtures/fakeCdpBrowser.ts +++ b/tests/fixtures/fakeCdpBrowser.ts @@ -143,6 +143,7 @@ export const startFakeCdpBrowser = async ( browserWebSocketUrl: `ws://127.0.0.1:${String(port)}/devtools/browser/fake`, allowedOrigin: endpoint, commands, + activeConnectionCount: () => sockets.size, emitRawMessage(message) { for (const socket of sockets) socket.send(message); }, diff --git a/tests/fixtures/fakeCdpBrowserTypes.ts b/tests/fixtures/fakeCdpBrowserTypes.ts index 54fc765632..e109178147 100644 --- a/tests/fixtures/fakeCdpBrowserTypes.ts +++ b/tests/fixtures/fakeCdpBrowserTypes.ts @@ -11,6 +11,8 @@ export interface FakeCdpBrowser { readonly browserWebSocketUrl: string; readonly allowedOrigin: string; readonly commands: readonly FakeCdpCommand[]; + /** Actual WebSocket connections still held by the wire fixture. */ + activeConnectionCount(): number; /** Deliver actual malformed wire data without bypassing the production transport parser. */ emitRawMessage(message: string): void; emitEvent(event: { diff --git a/tests/fixtures/fakeGhidra.mjs b/tests/fixtures/fakeGhidra.mjs index 5cc0f2c169..a1255a22c2 100644 --- a/tests/fixtures/fakeGhidra.mjs +++ b/tests/fixtures/fakeGhidra.mjs @@ -153,6 +153,16 @@ const handleRequest = (socket, server, request, state) => { const handlePing = (socket, request, state) => { state.pings.value += 1; + if (state.mode === "diagnostic_output" && state.pings.value === 1) { + const retainedTokenPrefixBytes = Math.floor(state.token.length / 2); + const preservedMarker = "source=/tmp/local-evidence.bin?cursor=keep&"; + const fillLength = 8 * 1024 * 1024 - retainedTokenPrefixBytes; + const fill = "x".repeat(fillLength - Buffer.byteLength(preservedMarker)); + process.stderr.write(`${fill}${preservedMarker}`); + setImmediate(() => { + process.stderr.write(state.token); + }); + } if (state.mode === "malformed") { socket.write("{invalid\n"); return; diff --git a/tests/fixtures/fakeHopper.mjs b/tests/fixtures/fakeHopper.mjs index 392f609c89..2c15dbd91f 100644 --- a/tests/fixtures/fakeHopper.mjs +++ b/tests/fixtures/fakeHopper.mjs @@ -1,7 +1,13 @@ import { createServer } from "node:net"; -const [socketPath, token, runId, shutdownMode = "acknowledge"] = - process.argv.slice(2); +const [ + socketPath, + token, + runId, + shutdownMode = "acknowledge", + diagnosticMode = "quiet", +] = process.argv.slice(2); +let diagnosticOutputEmitted = false; const heldReplies = new Map(); if (process.send !== undefined) process.on("message", (message) => { @@ -16,6 +22,18 @@ const shutdownResult = () => ({ document_closed: shutdownMode !== "unconfirmed", ...(shutdownMode === "cleanup-required" ? { cleanup_required: true } : {}), }); +const emitDiagnosticOutput = () => { + if (diagnosticMode === "quiet" || diagnosticOutputEmitted) return; + diagnosticOutputEmitted = true; + const retainedTokenPrefixBytes = Math.floor(token.length / 2); + const preservedMarker = "source=/tmp/local-evidence.bin?cursor=keep&"; + const fillLength = 8 * 1024 * 1024 - retainedTokenPrefixBytes; + const fill = "x".repeat(fillLength - Buffer.byteLength(preservedMarker)); + process.stderr.write(`${fill}${preservedMarker}`); + setImmediate(() => { + process.stderr.write(token); + }); +}; const capabilityUnavailableResult = (id) => ({ id, error: { @@ -182,6 +200,7 @@ const server = createServer((socket) => { if (request.token !== token) { send({ id: request.id, error: { code: -32001, message: "bad token" } }); } else if (request.method === "health") { + emitDiagnosticOutput(); send( { id: request.id, @@ -249,5 +268,6 @@ const closeServer = () => if (process.connected) process.disconnect(); }); -server.listen(socketPath); +if (diagnosticMode === "noisy_no_socket") emitDiagnosticOutput(); +else server.listen(socketPath); process.on("SIGTERM", closeServer); diff --git a/tests/fixtures/firmware/producer.mjs b/tests/fixtures/firmware/producer.mjs index 3c079c8315..6654c49410 100644 --- a/tests/fixtures/firmware/producer.mjs +++ b/tests/fixtures/firmware/producer.mjs @@ -1,5 +1,5 @@ import { createHash } from "node:crypto"; -import { mkdir, readFile, writeFile, symlink } from "node:fs/promises"; +import { chmod, mkdir, readFile, writeFile, symlink } from "node:fs/promises"; import { join } from "node:path"; const args = process.argv.slice(2); const mode = process.env.REA_FIRMWARE_FIXTURE_MODE; @@ -125,8 +125,19 @@ if (args.includes("--version")) { ]; await writeFile( reportPath, - mode === "malformed" ? "[" : JSON.stringify(records), + mode === "malformed" || mode === "workspace-cleanup-failure-malformed" + ? "[" + : JSON.stringify(records), ); if (mode === "dependency") process.exitCode = 1; + if ( + mode === "workspace-cleanup-failure" || + mode === "workspace-cleanup-failure-malformed" + ) { + const blocked = join(process.cwd(), "cleanup-blocked"); + await mkdir(blocked); + await writeFile(join(blocked, "retained"), "owned workspace bytes"); + await chmod(blocked, 0); + } } } diff --git a/tests/fixtures/firmware/provider.ts b/tests/fixtures/firmware/provider.ts index b6e32b9da7..5b73226c54 100644 --- a/tests/fixtures/firmware/provider.ts +++ b/tests/fixtures/firmware/provider.ts @@ -5,6 +5,7 @@ import { expect, onTestFinished } from "vitest"; import { createFirmwareAnalysisProvider } from "../../../src/composition/firmware.js"; import { FirmwareAnalysisService } from "../../../src/application/firmware/FirmwareAnalysisService.js"; import { spawnOwnedProviderProcess } from "../../../src/process/ProviderProcess.js"; +import { cleanupOwnedProcessGroup } from "../../../src/process/ProcessOwnership.js"; import { createTestTempDirectory } from "../temporaryDirectory.js"; /** Exercise real subprocess ownership with synthetic pinned producer representations. */ @@ -18,6 +19,7 @@ export const firmwareFixture = async (mode = "normal", banner?: string) => { pid: number | undefined; args: readonly string[]; }[] = []; + let cleanupUnverified = mode === "cleanup-failure"; const provider = createFirmwareAnalysisProvider( { ...process.env, @@ -54,21 +56,29 @@ export const firmwareFixture = async (mode = "normal", banner?: string) => { args: options.arguments, }); if (mode === "cleanup-failure") { - onTestFinished(async () => { - if (options.cwd !== undefined) - await rm(options.cwd, { recursive: true, force: true }); - }); return { ...spawned, - cleanup: async () => ({ - cleaned: false, - reason: "injected ownership failure", - }), + cleanup: () => + cleanupUnverified + ? Promise.resolve({ + cleaned: false, + reason: "injected ownership failure", + }) + : cleanupOwnedProcessGroup(spawned.ownership), }; } return spawned; }, ); + const close = async () => { + if (provider.close === undefined) + throw new Error("Expected firmware provider cleanup capability"); + await provider.close(); + }; + onTestFinished(async () => { + cleanupUnverified = false; + await close(); + }); return { provider, service: new FirmwareAnalysisService(provider), @@ -76,6 +86,10 @@ export const firmwareFixture = async (mode = "normal", banner?: string) => { output: join(root, "output"), root, launches, + close, + restoreCleanup: () => { + cleanupUnverified = false; + }, }; };