Merge latest main to include the verified uninstall fixture corrections

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
N0zoM1z0andKaoru0822 committed 2026-10-14 08:59:04 +00:00
commit 7a9991289c
77 files changed
+3984 -364

No files matched your search

+4 -3
View File
@@ -100,15 +100,16 @@ limits, security/privacy/containment implications, and meaningful proof gaps. --
The PR author is the owner by default. A different owner must explicitly accept
ownership on this PR.
AI agents must leave these declarations to a human to dictate to them or there could be catastrophic consequences for many users of REA.
Draft PRs may leave them unchecked. After further changes, clear the final checkbox
until the owner reviews the updated diff and refreshes affected validation.
Draft PRs may leave them unchecked. These declarations record initial readiness
when opening the PR for review; later pushes do not require a new human sign-off.
Keep the description and validation evidence current as the work changes.
See CONTRIBUTING.md#pull-request-ownership-and-readiness. -->
Human owner: <!-- @GitHub-handle -->
- [ ] I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
- [ ] I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
- [ ] I declare the current revision ready for maintainer review.
- [ ] I declare this PR ready for initial maintainer review.
## Before merge
+51
View File
@@ -0,0 +1,51 @@
name: PR readiness
on:
pull_request_target:
branches: [main, "release/*"]
types:
[
opened,
edited,
synchronize,
reopened,
ready_for_review,
converted_to_draft,
]
workflow_dispatch:
inputs:
pull_request_number:
description: Open PR to recheck
required: true
type: string
permissions: {}
concurrency:
group: pr-readiness-${{ github.event.pull_request.number || inputs.pull_request_number }}
cancel-in-progress: true
jobs:
readiness:
name: Validate author declaration
runs-on: ubuntu-24.04
timeout-minutes: 3
permissions:
contents: read
pull-requests: read
statuses: write
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
with:
ref: ${{ github.event.repository.default_branch }}
persist-credentials: false
- name: Check initial readiness
uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9
with:
script: |
const { pathToFileURL } = require("node:url");
const entry = pathToFileURL(
`${process.env.GITHUB_WORKSPACE}/scripts/ci/pr-readiness.mjs`,
);
const { runReadinessCheck } = await import(entry.href);
await runReadinessCheck({ github, context, core });
+6 -6
View File
@@ -94,7 +94,7 @@ required to open an issue.
Follow [human ownership and readiness](CONTRIBUTING.md#pull-request-ownership-and-readiness)
when preparing a PR. AI-assisted contributions are welcome; a human must review
and own the final diff, check the validation evidence and its limitations, and
explicitly declare the current revision ready for maintainer review.
explicitly declare the PR ready for initial maintainer review.
Keep the PR body concise and current, with the concrete problem, observable
outcome, approach, and evidence. Report only checks that actually ran as executed;
@@ -103,11 +103,11 @@ Address actionable automated review findings and explain disagreements.
Leave the template's **Author readiness** declarations unchecked for the human
owner to complete. Never attest to human review on their behalf. Keep the PR in
draft until the human declares the current diff ready. After further changes,
clear the ready-for-review checkbox and refresh affected evidence so the human
can review and renew the declaration. CI and automated review results support
review; they do not establish human readiness, maintainer scope approval, or
merge permission.
draft until the human declares the PR initially ready. Later pushes do not
invalidate that declaration or require another human sign-off. Keep the PR body
and affected validation current as work changes. The readiness check validates
the filled declaration; CI and automated review results do not establish that
human review occurred, maintainer scope approval, or merge permission.
<!-- BEGIN:turborepo-agent-rules -->
+18 -5
View File
@@ -67,7 +67,7 @@ Before requesting maintainer review, the human owner must:
passing. Follow the [verification requirements](#development-feedback-and-pr-verification)
for required checks and real-provider coverage.
- Complete the **Author readiness** declaration in the PR template, explicitly
stating that the current revision is ready for review.
stating that the PR is ready for initial review.
Keep the PR description concise and current. Explain the problem, observable
outcome, approach, and useful evidence; link long logs or put them in optional
@@ -78,10 +78,23 @@ approval or permission to merge.
Draft PRs may request early feedback with open questions and incomplete checks.
Leave the readiness declaration unchecked until the human owner has completed
it. After further changes, clear the ready-for-review checkbox until the owner
reviews the updated diff, refreshes affected validation and the PR description,
and renews the declaration. AI agents must leave human declarations to the human
owner and must not attest to a human review on their behalf.
it. The declaration records initial readiness; later pushes do not invalidate
it or require another human sign-off. Keep the PR description and affected
validation current as work changes. AI agents must leave human declarations to
the human owner and must not attest to a human review on their behalf.
The `PR readiness` check reports pending for drafts and requires a named human
owner and all three template declarations for non-drafts. It checks that the
declaration is filled, not whether the code or evidence is correct. Existing PRs
can copy the **Author readiness** section from the current template.
Open PRs must have distinct head commits because GitHub shares commit statuses.
Maintainers enable merge enforcement after the workflow is merged: run
`gh workflow run pr-readiness.yml -f pull_request_number=NUMBER` for existing
open PRs, then add `PR readiness` from GitHub Actions to the branch's required
status checks alongside its existing checks. Deploy the workflow to each
protected target branch before requiring it there. The workflow reads PR
metadata using trusted default-branch code and writes only a commit status.
Maintainers may return a PR to the author or close it when the problem is
unsupported, validation is missing, or the owner cannot explain the change.
+3
View File
@@ -26,6 +26,9 @@ Readiness is `rea inspect-adb-client`: it runs `adb version` and reports the
resolved binary, its version, and where it was selected from. A missing
binary reports `capability_unavailable` with the `REA_ADB_PATH` remediation
rather than attempting an installation.
The `binary_session` inventory uses the same bounded client-version probe; it
does not query devices, start the adb server, or establish that a requested
serial is reachable. Device state is checked by each device operation.
## Answer an analyst question
+2
View File
@@ -21,6 +21,8 @@ Readiness is `rea inspect-apktool-client`: it runs `apktool --version` and
reports the resolved launcher, its selection source, and its version. A
missing launcher reports `capability_unavailable` with the
`REA_APKTOOL_COMMAND` remediation rather than attempting an installation.
`binary_session` uses the same host probe for tool availability; it does not
validate a particular APK or guarantee that the selected input can be decoded.
On Windows, a `.bat` or `.cmd` launcher (such as Chocolatey's `apktool.bat`)
runs through an explicit `cmd.exe /d /s /c` call. That call has no AutoRun,
+12 -6
View File
@@ -639,12 +639,18 @@ memory has no original file offset. For FAT Mach-O, offsets refer to the origina
container file. Results retain Hopper's image-relative offset, the observed slice
base, and the source executable path.
Loader selection reads the Mach-O container header, including FAT files with a
single architecture. For FAT64, REA validates the architecture table and selected
Mach-O header, prepares a private thin image, and loads it with Hopper's native
Mach-O loader. It checks the entire source's SHA-256 while copying the slice;
source identity and reported offsets still refer to the original container.
An ambiguous architecture subtype requires an explicitly extracted thin image;
REA does not guess. Configured loader arguments remain explicit overrides.
single architecture. For FAT32 and FAT64 containers, REA validates the
architecture table and selected Mach-O header, prepares a private thin image,
and loads it with Hopper's native Mach-O loader instead of Hopper's implicit
FAT slice choice. For arm64 targets, REA ranks subtype bases 0 (plain arm64) and
2 (arm64e), ignoring capability bits for ranking while retaining the raw subtype
for slice identity. These slices' function-entry encodings decode under
`--aarch64`, ahead of Apple's arm64e.x1 variant (base subtype 12), which is
selected only when it is the arm64 family's only candidate. REA checks the
entire source's SHA-256 while copying the slice; source identity and reported
offsets still refer to the original container. Any other ambiguous architecture
subtype requires an explicitly extracted thin image; REA does not guess.
Configured loader arguments remain explicit overrides.
FAT64 is distinct from the CPU architecture: FAT32 containers can contain 64-bit
executable images. This preparation avoids the Raw Binary loader dialog observed
with native FAT64 loading on Hopper 6.1.0-demo.
+14 -3
View File
@@ -383,13 +383,24 @@ exit comparison like `settlement.elapsed_ms`.
Every capture requires `truncation_details`, with separate accounting for:
- `raw_terminal`: original UTF-8 PTY chunk bytes and observed/retained chunk
counts. A chunk that does not fit `limits.output_bytes` is omitted whole;
a later smaller chunk can still fit.
counts. A chunk that does not fit either its remaining original-text budget
or normalized-text budget is omitted whole; a later smaller chunk can still
fit. A normalized-text omission also omits its original chunk, so raw and
normalized coverage can both be incomplete for that event.
- `normalized_terminal`: normalized comparison-text bytes retained from raw
chunks. This is optional when reading historical captures. A raw-fitting
chunk is omitted if an ordered normalization stage would exceed the remaining
`output_bytes` budget before the final normalized size is known. Its final
size is unknown, not assumed to equal the budget. The capture records the
omitted-frame count and a limitation; later smaller chunks remain eligible.
- `rendered_terminal`: cumulative UTF-8 serialized-state and visible-line
bytes, plus observed/retained frame counts. This is independent of the raw
chunk budget and does not count JSON encoding. Repeated screen snapshots can
exhaust it even when all raw output fits. Rendered states use retained PTY
input, so raw omissions also limit rendered coverage.
input, so raw omissions also limit rendered coverage. When bounded
normalization omits a rendered frame before its final size is known,
`unknown_size_frames` reports that count; `observed_bytes` includes only
rendered frame sizes that were determined exactly.
- `filesystem_before` and `filesystem_after`: file-count/depth limits,
enumeration failures, whole-file hash budget and bytes successfully hashed.
Directory names are read incrementally within the remaining entry capacity.
+1
View File
@@ -10,6 +10,7 @@ const HOST_TEST_OWNERS = ["apple", "package", "sqlite"];
// workflows' release, publication and self-hosted runner safeguards.
const GUARDED_WORKFLOWS = new Set([
".github/workflows/pr-title.yml",
".github/workflows/pr-readiness.yml",
".github/workflows/ci-full-pr.yml",
".github/workflows/release.yml",
".github/workflows/real-ghidra-windows.yml",
+63
View File
@@ -0,0 +1,63 @@
export interface ReadinessPullRequest {
readonly number: number;
readonly state: "open" | "closed";
readonly draft: boolean;
readonly body: string | null;
readonly head: { readonly sha: string };
readonly base: { readonly ref: string };
}
export interface ReadinessResult {
readonly state: "success" | "failure" | "pending";
readonly description: string;
}
export interface ReadinessGitHub {
readonly rest: {
readonly pulls: {
get(input: {
owner: string;
repo: string;
pull_number: number;
}): Promise<{ data: ReadinessPullRequest }>;
};
readonly repos: {
createCommitStatus(
input: ReadinessResult & {
owner: string;
repo: string;
sha: string;
context: string;
target_url: string;
},
): Promise<unknown>;
readonly listPullRequestsAssociatedWithCommit: unknown;
};
};
paginate(
method: unknown,
input: {
owner: string;
repo: string;
commit_sha: string;
per_page: number;
},
): Promise<
readonly Pick<ReadinessPullRequest, "number" | "state" | "head">[]
>;
}
export function evaluateReadiness(
pull: Pick<ReadinessPullRequest, "body" | "draft">,
template: string,
): ReadinessResult;
export function runReadinessCheck(input: {
github: ReadinessGitHub;
context: {
repo: { owner: string; repo: string };
payload: {
pull_request?: { number: number };
inputs?: { pull_request_number?: string };
};
serverUrl: string;
runId: number;
};
core: { info(message: string): void; setFailed(message: string): void };
}): Promise<ReadinessResult | undefined>;
+173
View File
@@ -0,0 +1,173 @@
import { readFile } from "node:fs/promises";
const result = (state, description) => ({ state, description });
function readinessLines(markdown) {
const lines = markdown.split(/\r?\n/u);
const visible = [];
let fence;
let comment = false;
for (const source of lines) {
if (fence) {
if (
new RegExp(
`^ {0,3}${fence.character}{${fence.length},}\\s*$`,
"u",
).test(source)
)
fence = undefined;
continue;
}
const opening = !comment && /^ {0,3}(`{3,}|~{3,})/u.exec(source);
if (opening) {
fence = { character: opening[1][0], length: opening[1].length };
continue;
}
let line = "";
let offset = 0;
while (offset < source.length) {
const marker = comment ? "-->" : "<!--";
const index = source.indexOf(marker, offset);
if (!comment)
line += source.slice(offset, index === -1 ? source.length : index);
if (index === -1) break;
offset = index + marker.length;
comment = !comment;
}
visible.push(line);
}
const starts = visible.flatMap((line, index) =>
/^## Author readiness\s*$/u.test(line) ? [index] : [],
);
if (starts.length !== 1) return [];
const remaining = visible.slice(starts[0] + 1);
const end = remaining.findIndex((line) => /^#{1,2}\s/u.test(line));
return (end === -1 ? remaining : remaining.slice(0, end)).map((line) =>
line.trimEnd(),
);
}
/** Check the initial declaration against the trusted repository template. */
export function evaluateReadiness(pull, template) {
if (pull.draft)
return result(
"pending",
"Draft PR: complete Author readiness before requesting review.",
);
const expected = readinessLines(template).flatMap((line) => {
const match = /^- \[ \] (.+)$/u.exec(line);
return match ? [match[1]] : [];
});
if (expected.length !== 3)
throw new Error(
"Trusted PR template must contain three readiness declarations.",
);
const lines = readinessLines(pull.body ?? "");
const owners = lines.filter((line) => line.startsWith("Human owner:"));
if (
owners.length !== 1 ||
!/^Human owner: @[a-z\d](?:[a-z\d-]*[a-z\d])?$/iu.test(owners[0])
)
return result(
"failure",
"Fill Human owner with one @GitHub-handle in the Author readiness section.",
);
const declarations = lines.flatMap((line) => {
const match = /^- \[([ xX])\] (.+)$/u.exec(line);
return match
? [{ checked: match[1].toLowerCase() === "x", text: match[2] }]
: [];
});
if (
expected.some((text) => {
const matches = declarations.filter((item) => item.text === text);
return matches.length !== 1 || !matches[0].checked;
})
)
return result(
"failure",
"Complete all three Author readiness declarations from the current PR template.",
);
return result(
"success",
"Human owner and all three initial readiness declarations are filled.",
);
}
/** Publish readiness on the PR head without running contributor code. */
export async function runReadinessCheck({ github, context, core }) {
const number =
context.payload.pull_request?.number ??
Number(context.payload.inputs?.pull_request_number);
if (!Number.isSafeInteger(number) || number <= 0)
throw new Error("A positive pull request number is required.");
const repository = context.repo;
const getPull = async () =>
(await github.rest.pulls.get({ ...repository, pull_number: number })).data;
const pull = await getPull();
if (pull.state !== "open" || !/^(?:main|release\/.+)$/u.test(pull.base.ref))
return;
const publish = (sha, verdict) =>
github.rest.repos.createCommitStatus({
...repository,
sha,
context: "PR readiness",
...verdict,
target_url: `${context.serverUrl}/${repository.owner}/${repository.repo}/actions/runs/${context.runId}`,
});
await publish(
pull.head.sha,
result("pending", "Checking the initial author readiness declaration."),
);
const template = await readFile(
new URL("../../.github/pull_request_template.md", import.meta.url),
"utf8",
);
let verdict = evaluateReadiness(pull, template);
if (verdict.state === "success") {
const associated = await github.paginate(
github.rest.repos.listPullRequestsAssociatedWithCommit,
{ ...repository, commit_sha: pull.head.sha, per_page: 100 },
);
if (
!associated.some(
(other) =>
other.number === number &&
other.state === "open" &&
other.head.sha === pull.head.sha,
)
)
verdict = result(
"pending",
"GitHub has not confirmed this PR's head association; rerun PR readiness.",
);
else if (
associated.some(
(other) =>
other.state === "open" &&
other.number !== number &&
other.head.sha === pull.head.sha,
)
)
verdict = result(
"failure",
"Open PRs must have distinct head commits because GitHub shares commit statuses.",
);
}
const latest = await getPull();
if (latest.state !== "open") return;
if (
latest.head.sha !== pull.head.sha ||
latest.body !== pull.body ||
latest.draft !== pull.draft ||
latest.base.ref !== pull.base.ref
)
verdict = result(
"pending",
"PR changed during the check; wait for the latest run or rerun PR readiness.",
);
await publish(latest.head.sha, verdict);
core.info(verdict.description);
if (verdict.state === "failure") core.setFailed(verdict.description);
return verdict;
}
+8
View File
@@ -55,4 +55,12 @@ describe("parseAdbVersionOutput", () => {
installedPath: null,
});
});
it("does not accept arbitrary successful output as an adb version", () => {
expect(
parseAdbVersionOutput(
"Android Debug Bridge version tomorrow\nVersion unknown\n",
),
).toMatchObject({ version: null });
});
});
+30 -8
View File
@@ -67,7 +67,9 @@ export interface AdbClientIdentity {
* `Version ` the build version (no colon), and `Installed as` the binary path.
*/
export const parseAdbVersionOutput = (stdout: string): AdbClientIdentity => {
const build = /^Version\s+(\S.*)$/mu.exec(stdout);
const build = /^Version\s+(\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?)\s*$/mu.exec(
stdout,
);
const installed = /^Installed as\s+(.+)$/mu.exec(stdout);
return {
version: build === null ? null : build[1]!.trim(),
@@ -84,14 +86,34 @@ export const inspectAdbClient = async (
signal?: AbortSignal,
): Promise<AdbClientIdentity> => {
try {
const { stdout } = await execFileOutput(selection.binary, ["version"], {
timeout: VERSION_TIMEOUT_MS,
stopSignal: "SIGTERM",
maxBuffer: VERSION_MAX_OUTPUT_BYTES,
...(signal === undefined ? {} : { signal }),
});
return parseAdbVersionOutput(stdout);
const { stdout, stderr } = await execFileOutput(
selection.binary,
["version"],
{
timeout: VERSION_TIMEOUT_MS,
stopSignal: "SIGTERM",
maxBuffer: VERSION_MAX_OUTPUT_BYTES,
...(signal === undefined ? {} : { signal }),
},
);
const identity = parseAdbVersionOutput(stdout);
const hasRecognizedProtocolVersion =
/^Android Debug Bridge version\s+\d+\.\d+\.\d+\s*$/mu.test(stdout);
if (identity.version === null && !hasRecognizedProtocolVersion)
throw new AdbConfigurationFailure(
"version_unresolved",
"The selected adb binary exited successfully but did not report a recognizable protocol or build version",
{
selected_binary: selection.binary,
path_source: selection.pathSource,
version_probe: "unrecognized_output",
stdout,
stderr,
},
);
return identity;
} catch (cause) {
if (cause instanceof AdbConfigurationFailure) throw cause;
const failure = execFileOutputFailure(cause);
if (signal?.aborted || failure?.code === "ABORT_ERR")
throw new AnalysisCancelledError("inspect_adb_client", { cause });
+30
View File
@@ -91,6 +91,36 @@ it("inspects the adb client without contacting the server", async () => {
expect(result.contacted_adb_server).toBe(false);
});
it("fails closed when adb exits successfully without a recognized version", async () => {
const stdout = `unexpected adb output ${"x".repeat(700)}\n`;
const stderr = `probe stderr ${"y".repeat(700)}\n`;
const { binary } = await writeAdbStub(
() => `printf '%s' '${stdout}'; printf '%s' '${stderr}' >&2; exit 0`,
);
await expect(
providerOver(binary).inspectAvailability(),
).resolves.toMatchObject({
status: "unavailable",
code: "version_unresolved",
diagnostics: {
version_probe: "unrecognized_output",
stdout,
stderr,
},
});
});
it("accepts adb's recognized protocol version when the build version is absent", async () => {
const { binary } = await writeAdbStub(
() => `printf 'Android Debug Bridge version 1.0.41\\n'`,
);
await expect(
providerOver(binary).inspectAvailability(),
).resolves.toMatchObject({
status: "available",
});
});
it("lists devices with inferred kinds and observed daemon startup", async () => {
const { binary } = await writeAdbStub(
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
+22 -1
View File
@@ -58,7 +58,28 @@ describe("parseApktoolVersionOutput", () => {
});
it("reports null when the launcher prints nothing usable", () => {
expect(parseApktoolVersionOutput(" \n")).toEqual({ version: null });
expect(parseApktoolVersionOutput(" \n")).toEqual({
version: null,
reportedVersion: null,
});
expect(parseApktoolVersionOutput("successful but not a version\n")).toEqual(
{
version: null,
reportedVersion: "successful but not a version",
},
);
});
it("parses semantic prerelease versions without claiming they meet the minimum", () => {
expect(parseApktoolVersionOutput("2.7.0-beta.1\n")).toEqual({
version: "2.7.0-beta.1",
});
});
it("validates normalized semantic versions while preserving caller spelling", () => {
expect(parseApktoolVersionOutput("v2.7.0\n")).toEqual({
version: "v2.7.0",
});
});
});
+52 -10
View File
@@ -15,6 +15,7 @@ import {
import type { ProviderAvailability } from "../application/AnalysisProvider.js";
import type { ProviderRejectionCode } from "../contracts/providerSelection.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { gte, valid } from "semver";
import {
isWindowsBatchFile,
windowsBatchCommand,
@@ -148,7 +149,8 @@ export type ApktoolConfigurationFailureCode =
| "command_missing"
| "command_unresolved"
| "launcher_unsupported"
| "version_unresolved";
| "version_unresolved"
| "unsupported_version";
export class ApktoolConfigurationFailure extends Error {
constructor(
@@ -161,9 +163,9 @@ export class ApktoolConfigurationFailure extends Error {
}
}
export interface ApktoolClientIdentity {
readonly version: string | null;
}
export type ApktoolClientIdentity =
| { readonly version: string; readonly reportedVersion?: never }
| { readonly version: null; readonly reportedVersion: string | null };
/** Process creation for one apktool run on the selected host. */
export interface ApktoolInvocation {
@@ -199,7 +201,7 @@ export const apktoolInvocation = (
windowsVerbatimArguments: false,
};
/** Read the version from `apktool --version` output (first non-empty line). */
/** Read the version from Apktool's single-line `--version` response. */
export const parseApktoolVersionOutput = (
stdout: string,
): ApktoolClientIdentity => {
@@ -207,7 +209,17 @@ export const parseApktoolVersionOutput = (
.split(/\r?\n/u)
.map((entry) => entry.trim())
.find((entry) => entry !== "");
return { version: line === undefined ? null : line };
if (line === undefined || valid(line) === null)
return { version: null, reportedVersion: line ?? null };
return { version: line };
};
const supportsDocumentedApktoolVersion = (version: string): boolean => {
// Debian's verified 2.7.0-dirty build is a downstream build marker, not a
// prerelease. Treat only that known marker as build metadata for the floor.
const comparisonVersion = version.replace(/-dirty$/u, "+dirty");
const parsedVersion = valid(comparisonVersion);
return parsedVersion !== null && gte(parsedVersion, "2.7.0");
};
/** Probe the selected apktool launcher without decoding anything. */
@@ -239,8 +251,10 @@ export const inspectApktoolClient = async (
{ cause },
);
}
let stdout: string;
let stderr: string;
try {
const { stdout } = await execFileOutput(
({ stdout, stderr } = await execFileOutput(
invocation.command,
invocation.arguments,
{
@@ -251,8 +265,7 @@ export const inspectApktoolClient = async (
env: hostEnvironment,
...(signal === undefined ? {} : { signal }),
},
);
return parseApktoolVersionOutput(stdout);
));
} catch (cause) {
const failure = execFileOutputFailure(cause);
if (signal?.aborted || failure?.code === "ABORT_ERR")
@@ -283,6 +296,33 @@ export const inspectApktoolClient = async (
{ cause },
);
}
const identity = parseApktoolVersionOutput(stdout);
if (identity.version === null)
throw new ApktoolConfigurationFailure(
"version_unresolved",
"The selected apktool launcher exited successfully but did not report a recognizable version",
{
selected_command: selection.command,
command_source: selection.commandSource,
version_probe: "unrecognized_output",
reported_version: identity.reportedVersion,
stdout,
stderr,
},
);
if (!supportsDocumentedApktoolVersion(identity.version))
throw new ApktoolConfigurationFailure(
"unsupported_version",
`The selected Apktool version ${identity.version} is older than the supported minimum 2.7.0`,
{
selected_command: selection.command,
command_source: selection.commandSource,
reported_version: identity.version,
minimum_version: "2.7.0",
remediation: `Use Apktool 2.7.0 or newer; see docs/apktool-resource-analysis.md.`,
},
);
return identity;
};
/** Availability in the shape shared by every provider. */
@@ -295,7 +335,9 @@ const rejectionCodeOf = (
code === "launcher_unsupported" ||
code === "command_unresolved"
? "not_configured"
: "version_unresolved";
: code === "unsupported_version"
? "unsupported_version"
: "version_unresolved";
export const apktoolConfigurationAvailability = (
cause: ApktoolConfigurationFailure,
+35 -2
View File
@@ -17,7 +17,7 @@ import type { ApktoolRequest } from "../domain/apktool/apktoolResourceAnalysis.j
const it = test.skipIf(process.platform === "win32");
const VERSION_BLOCK = "printf '2.7.0-stub\\n'";
const VERSION_BLOCK = "printf '2.7.0-dirty\\n'";
const REAL_YML = [
"!!brut.androlib.meta.MetaInfo",
@@ -172,7 +172,7 @@ it("inspects the apktool launcher", async () => {
expect(result.client).toEqual({
command,
command_source: "environment",
apktool_version: "2.7.0-stub",
apktool_version: "2.7.0-dirty",
});
});
@@ -355,6 +355,39 @@ it("reports a missing launcher as a capability gap with recovery", async () => {
});
});
it("fails closed for unrecognized and unsupported Apktool versions", async () => {
const invalidOutput = `unexpected Apktool output ${"x".repeat(700)}\n`;
const unrecognized = await writeApktoolStub(
() => `printf '%s' '${invalidOutput}'; exit 0`,
);
await expect(
providerOver(unrecognized.command).inspectAvailability(),
).resolves.toMatchObject({
status: "unavailable",
code: "version_unresolved",
diagnostics: {
reported_version: invalidOutput.trim(),
stdout: invalidOutput,
},
});
const old = await writeApktoolStub(() => `printf '2.6.3\\n'`);
await expect(
providerOver(old.command).inspectAvailability(),
).resolves.toMatchObject({
status: "unavailable",
code: "unsupported_version",
});
const prerelease = await writeApktoolStub(() => `printf '2.7.0-beta.1\\n'`);
await expect(
providerOver(prerelease.command).inspectAvailability(),
).resolves.toMatchObject({
status: "unavailable",
code: "unsupported_version",
});
});
it("rejects a target that is not a readable file", async () => {
const { command, root } = await writeApktoolStub(decodeStub());
const projected = await refusedBy(providerOver(command), {
@@ -97,6 +97,53 @@ const entry = (
return found;
};
describe("ADB and Apktool host availability", () => {
it("uses only their selected owner probes", () => {
const ready = {
status: "available" as const,
code: null,
reason: null,
diagnostics: {},
};
const configuredButInvalid = {
status: "unavailable" as const,
code: "not_configured" as const,
reason: "REA_ADB_PATH is invalid",
diagnostics: {},
};
const readyInventory = buildCapabilityInventory(status(), {
...enabledPolicy,
adbAnalysisAvailability: ready,
apktoolAnalysisAvailability: ready,
});
expect(
readyInventory.find(({ name }) => name === "inspect_adb_client"),
).toMatchObject({ available: true, reason: "available" });
expect(
readyInventory.find(({ name }) => name === "inspect_apktool_client"),
).toMatchObject({ available: true, reason: "available" });
const unavailableInventory = buildCapabilityInventory(status(), {
...enabledPolicy,
adbAnalysisAvailability: configuredButInvalid,
apktoolAnalysisAvailability: {
status: "unavailable",
code: "unsupported_version",
reason: "Apktool 2.6.3 is older than 2.7.0",
diagnostics: {},
},
});
expect(
unavailableInventory.find(({ name }) => name === "inspect_adb_client"),
).toMatchObject({ available: false, reason: "provider_missing" });
expect(
unavailableInventory.find(
({ name }) => name === "decode_android_resources",
),
).toMatchObject({ available: false, reason: "provider_unavailable" });
});
});
describe("capability failure reasons", () => {
it("keeps an unavailable Android runtime distinct from a missing provider", () => {
const inventory = buildCapabilityInventory(status(), {
+31
View File
@@ -54,6 +54,8 @@ export type AvailabilityPolicy = {
readonly firmwareInspectionEnabled?: boolean;
readonly firmwareExtractionEnabled?: boolean;
readonly androidAnalysisAvailability?: ProviderAvailability;
readonly adbAnalysisAvailability?: ProviderAvailability;
readonly apktoolAnalysisAvailability?: ProviderAvailability;
readonly jebAnalysisAvailability?: ProviderAvailability;
readonly javascriptRecoveryEnabled?: boolean;
readonly webModuleResolutionEnabled?: boolean;
@@ -473,6 +475,16 @@ const workflowAvailabilityFor = ({
policy.androidAnalysisAvailability?.reason ??
"Set REA_JADX_MCP_JAR to a caller-supplied jadx-headless-mcp 0.7.1 JAR and provide a full JDK on Linux, macOS, or Windows x64 with its matching bundled native controls.",
};
if (kind === "adb-provider")
return ownerProbeAvailability(
policy.adbAnalysisAvailability,
"Set REA_ADB_PATH to a caller-supplied adb executable or make adb available on PATH. This checks the client version only; device and serial availability depend on the request.",
);
if (kind === "apktool-provider")
return ownerProbeAvailability(
policy.apktoolAnalysisAvailability,
"Set REA_APKTOOL_COMMAND to a caller-supplied Apktool executable or make Apktool 2.7.0 or newer available on PATH.",
);
if (kind === "jeb-provider")
return policy.jebAnalysisAvailability?.status === "available"
? { reason: "available", remediation: null }
@@ -506,6 +518,25 @@ const workflowAvailabilityFor = ({
return null;
};
const ownerProbeAvailability = (
observation: ProviderAvailability | undefined,
missingRemediation: string,
): Availability => {
if (observation === undefined)
return { reason: "provider_missing", remediation: missingRemediation };
if (observation.status === "available")
return { reason: "available", remediation: null };
return {
reason:
observation.code === "unsupported_host"
? "unsupported_host"
: observation.code === "not_configured"
? "provider_missing"
: "provider_unavailable",
remediation: observation.reason,
};
};
const electronProviderAvailability = (
name: string,
kind: ToolKind,
+147 -42
View File
@@ -47,6 +47,49 @@ import { resolveProcedureAddress } from "./ProcedureAddressResolution.js";
import { traceLiteralFeature } from "./EnhancedLiteralTracing.js";
import { projectNativeApiInspection } from "./native/NativeApiInspection.js";
type BatchProcedureIdentity =
| {
readonly status: "resolved";
readonly address: string;
readonly name: string | null;
readonly name_error?: ReturnType<typeof projectAnalysisError>;
}
| {
readonly status: "unknown";
readonly error: ReturnType<typeof projectAnalysisError>;
};
type BatchResolution = {
readonly address: string;
readonly result: Result<string, AnalysisError>;
};
type BatchReadyItem = {
readonly address: string;
readonly procedure: BatchProcedureIdentity;
readonly providerProcedure: string;
};
type BatchDecompileItem = {
readonly address: string;
readonly procedure: BatchProcedureIdentity;
readonly status: "ok" | "error";
readonly pseudocode?: string;
readonly error?: ReturnType<typeof projectAnalysisError>;
};
const isAnalysisCancellation = (
error: AnalysisError,
): error is AnalysisCancelledError => error._tag === "AnalysisCancelledError";
const firstAnalysisCancellation = <Value>(
results: readonly Result<Value, AnalysisError>[],
): AnalysisCancelledError | undefined => {
for (const result of results)
if (!result.ok && isAnalysisCancellation(result.error)) return result.error;
return undefined;
};
/**
* Composes direct provider operations into bounded reverse-engineering tools.
* Direct callers use `execute`; adapters that own validation use
@@ -259,52 +302,34 @@ export class EnhancedTools {
addresses: readonly string[],
signal?: AbortSignal,
): EnhancedResult {
const items = await Promise.all(
const resolutions = await Promise.all(
addresses.map(async (address) => {
const resolved = await resolveProcedureAddress(
const result = await resolveProcedureAddress(
this.#call.bind(this),
address,
signal,
);
const procedure = resolved.ok
? await this.#batchProcedureIdentity(resolved.value, signal)
: {
status: "unknown" as const,
error: projectAnalysisError(resolved.error),
};
const result = await this.#call(
"procedure_pseudo_code",
{ procedure: resolved.ok ? resolved.value : address },
signal,
resolved.ok ? "provider" : undefined,
);
if (!result.ok)
return {
address,
procedure,
status: "error" as const,
error: projectAnalysisError(result.error),
};
if (typeof result.value !== "string" || result.value.length === 0)
return {
address,
procedure,
status: "error" as const,
error: projectAnalysisError(
new AnalysisOutputError(
"procedure_pseudo_code",
"provider returned empty pseudocode",
),
),
};
return {
address,
procedure,
status: "ok" as const,
pseudocode: result.value,
};
return { address, result };
}),
);
const resolutionCancellation = firstAnalysisCancellation(
resolutions.map(({ result }) => result),
);
if (resolutionCancellation !== undefined)
return err(resolutionCancellation);
const identities = await this.#batchIdentities(resolutions, signal);
const ready: BatchReadyItem[] = [];
for (const identity of identities) {
if (!identity.ok) return err(identity.error);
ready.push(identity.value);
}
const decompilations = await this.#batchPseudocode(ready, signal);
const items: BatchDecompileItem[] = [];
for (const decompilation of decompilations) {
if (!decompilation.ok) return err(decompilation.error);
items.push(decompilation.value);
}
const succeeded = items.filter(({ status }) => status === "ok").length;
return ok({
items,
@@ -314,12 +339,91 @@ export class EnhancedTools {
});
}
async #batchProcedureIdentity(address: string, signal?: AbortSignal) {
async #batchIdentities(
resolutions: readonly BatchResolution[],
signal?: AbortSignal,
): Promise<Result<BatchReadyItem, AnalysisCancelledError>[]> {
return Promise.all(
resolutions.map(async ({ address, result }) => {
if (!result.ok)
return ok({
address,
providerProcedure: address,
procedure: {
status: "unknown" as const,
error: projectAnalysisError(result.error),
},
});
const identity = await this.#batchProcedureIdentity(
result.value,
signal,
);
return identity.ok
? ok({
address,
providerProcedure: result.value,
procedure: identity.value,
})
: err(identity.error);
}),
);
}
async #batchPseudocode(
ready: readonly BatchReadyItem[],
signal?: AbortSignal,
): Promise<Result<BatchDecompileItem, AnalysisCancelledError>[]> {
return Promise.all(
ready.map(async ({ address, procedure, providerProcedure }) => {
const result = await this.#call(
"procedure_pseudo_code",
{ procedure: providerProcedure },
signal,
procedure.status === "resolved" ? "provider" : undefined,
);
if (!result.ok && isAnalysisCancellation(result.error))
return err(result.error);
const item: BatchDecompileItem = !result.ok
? {
address,
procedure,
status: "error",
error: projectAnalysisError(result.error),
}
: typeof result.value !== "string" || result.value.length === 0
? {
address,
procedure,
status: "error",
error: projectAnalysisError(
new AnalysisOutputError(
"procedure_pseudo_code",
"provider returned empty pseudocode",
),
),
}
: {
address,
procedure,
status: "ok",
pseudocode: result.value,
};
return ok(item);
}),
);
}
async #batchProcedureIdentity(
address: string,
signal?: AbortSignal,
): Promise<Result<BatchProcedureIdentity, AnalysisCancelledError>> {
const observed = await this.#call("address_name", { address }, signal);
if (!observed.ok && isAnalysisCancellation(observed.error))
return err(observed.error);
const named =
observed.ok &&
(typeof observed.value === "string" || observed.value === null);
return {
return ok({
status: "resolved" as const,
address,
name: named ? observed.value : null,
@@ -335,7 +439,7 @@ export class EnhancedTools {
: observed.error,
),
}),
};
});
}
async #callGraph(
@@ -385,6 +489,7 @@ export class EnhancedTools {
"provider",
);
if (!result.ok) {
if (isAnalysisCancellation(result.error)) return result;
graph[level].push({
address: current.address,
status: "error",
@@ -28,6 +28,7 @@ it("retains batched TraceMap-resolved paths and null source positions", () => {
text: { included: true, value: sourceMapText },
},
],
unavailable_relevant_files: [],
containers: [],
text_bytes_read: Buffer.byteLength(sourceMapText),
invalid_utf8_files: 0,
@@ -178,6 +178,7 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
);
const unknownGap =
(context.fileSet.source_selection?.unselected_files ?? 0) > 0 ||
context.fileSet.unavailable_relevant_files.length > 0 ||
context.analysis.files.some(
({ analysis_failure, application_projection_failure }) =>
analysis_failure !== undefined ||
@@ -1,7 +1,10 @@
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { ApplicationNode } from "../../domain/javascript/javascriptApplicationGraphSchemas.js";
import { htmlInlineScriptLimitation } from "./JavaScriptArtifactAnalysis.js";
import { completeApplicationCoverage } from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import {
completeApplicationCoverage,
partialApplicationCoverage,
} from "../../domain/javascript/javascriptApplicationEvidenceSchemas.js";
import type { JavaScriptModuleArtifactAnalysis } from "./JavaScriptArtifactAnalysisTypes.js";
import { jsonValueSchema } from "../../domain/jsonValue.js";
import { freezeJsonSnapshot } from "../../domain/immutableJson.js";
@@ -300,8 +303,133 @@ export const addJavaScriptArtifactFiles = (
},
});
}
addUnavailableAsarEntries(context);
};
const addUnavailableAsarEntries = (
context: JavaScriptArtifactGraphContext,
): void => {
for (const file of context.fileSet.unavailable_relevant_files) {
if (!isAsarContainer(context, file.container_sha256)) continue;
const limitations = [
...(file.limitations ?? []),
`The inventoried ${file.kind ?? "relevant"} entry has unavailable bytes; no content artifact identity can be established.`,
];
const entry = context.accumulator.addNode({
kind: "asar-entry",
identity: {
strategy: "canonical-path",
stability: "artifact-version",
artifact_sha256: file.container_sha256,
path: file.path,
},
observations: [
{
label: file.path,
properties: {
entry_sha256: file.sha256,
bytes: file.bytes,
inventory_artifact_id: file.inventory_artifact_id,
...(file.inventory_occurrence_id === undefined
? {}
: { inventory_occurrence_id: file.inventory_occurrence_id }),
hash_status: file.hash_status,
artifact_kind: file.artifact_kind,
...(file.kind === undefined ? {} : { kind: file.kind }),
content_available: false,
},
evidence: artifactObservationEvidence({
sha256: file.container_sha256,
path: file.path,
operation: "inventory-asar-unavailable-entry",
coverage: partialApplicationCoverage([], null),
limitations,
}),
},
],
});
const parent =
context.containerNodes.get(file.container_sha256) ?? context.root;
context.accumulator.addEdge({
source_node_id: parent.node_id,
target_node_id: entry.node_id,
relation: "contains",
properties: {
path: file.path,
content_available: false,
hash_status: file.hash_status,
},
evidence: artifactObservationEvidence({
sha256: file.container_sha256,
path: file.path,
operation: "inventory-asar-unavailable-entry",
coverage: partialApplicationCoverage([], null),
limitations,
}),
});
if (file.kind === "native-addon" || file.artifact_kind === "native-addon") {
const addon = context.accumulator.addNode({
kind: "native-addon",
identity: {
strategy: "canonical-path",
stability: "artifact-version",
artifact_sha256: file.container_sha256,
path: file.path,
},
observations: [
{
label: file.path,
properties: {
availability: "unavailable",
...(file.hash_status === undefined
? {}
: { hash_status: file.hash_status }),
...(file.artifact_kind === undefined
? {}
: { artifact_kind: file.artifact_kind }),
bytes: file.bytes,
inventory_artifact_id: file.inventory_artifact_id,
...(file.inventory_occurrence_id === undefined
? {}
: { inventory_occurrence_id: file.inventory_occurrence_id }),
},
evidence: artifactObservationEvidence({
sha256: file.container_sha256,
path: file.path,
operation: "classify-unavailable-native-addon",
coverage: partialApplicationCoverage([], null),
limitations,
}),
},
],
});
context.accumulator.addEdge({
source_node_id: entry.node_id,
target_node_id: addon.node_id,
relation: "contains",
properties: {
classification: "native-addon",
content_available: false,
},
evidence: artifactObservationEvidence({
sha256: file.container_sha256,
path: file.path,
operation: "classify-unavailable-native-addon",
coverage: partialApplicationCoverage([], null),
limitations,
}),
});
}
}
};
const isAsarContainer = (
context: JavaScriptArtifactGraphContext,
containerSha256: string,
): boolean =>
context.snapshot.manifest.root_format === "asar" ||
containerSha256 !== context.snapshot.manifest.root_sha256;
const sourceFileRange = (
file: JavaScriptArtifactFile,
): JavaScriptSourceRange => {
@@ -464,10 +592,7 @@ const createAsarEntry = (
context: JavaScriptArtifactGraphContext,
file: JavaScriptArtifactFile,
): ApplicationNode | undefined => {
const isAsar =
context.snapshot.manifest.root_format === "asar" ||
file.container_sha256 !== context.snapshot.manifest.root_sha256;
if (!isAsar) return undefined;
if (!isAsarContainer(context, file.container_sha256)) return undefined;
return context.accumulator.addNode({
kind: "asar-entry",
identity: {
@@ -68,6 +68,56 @@ describe("artifact-local graph identity", () => {
});
describe("loader-owned path lookup", () => {
it("applies URL whitespace handling only to ESM lookup and retains declarations", () => {
const tab = "\t";
const files = fileMap([
file("app/entry.mjs", "root"),
file("app/feature.js", "root"),
file(`app/tab${tab}name.js`, "root"),
]);
for (const whitespace of ["\t", "\n", "\r"]) {
const raw = `./fea${whitespace}ture.js`;
expect(
resolve({
declaredPath: raw,
sourcePath: "app/entry.mjs",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ declared_path: raw, resolved_path: "app/feature.js" });
}
expect(
resolve({
declaredPath: `./tab${tab}name.js`,
sourcePath: "app/entry.mjs",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ resolved_path: null });
expect(
resolve({
declaredPath: `./tab${tab}name.js`,
sourcePath: "app/entry.cjs",
context: "module-specifier",
moduleKind: "require",
files,
}),
).toMatchObject({ resolved_path: "app/tab\tname.js" });
expect(
resolve({
declaredPath: "./tab%09name.js",
sourcePath: "app/entry.mjs",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ resolved_path: "app/tab\tname.js" });
});
});
describe("loader-specific module suffix lookup", () => {
it("keeps relative ESM imports exact", () => {
const files = fileMap([
file("app/entry.mjs", "root"),
@@ -186,6 +236,42 @@ describe("loader-owned path lookup", () => {
});
});
describe("unavailable inventoried native add-ons", () => {
it("retains an inventoried native add-on whose bytes are unavailable", () => {
const unavailable = {
path: "app/addon.node",
container_sha256: file("app/entry.cjs", "root").container_sha256,
sha256: null,
bytes: 8192,
inventory_artifact_id: null,
inventory_occurrence_id: "missing-addon-occurrence",
kind: "native-addon" as const,
hash_status: "unavailable" as const,
limitations: ["unpacked companion is missing"],
};
const result = resolve({
declaredPath: "./addon.node",
sourcePath: "app/entry.cjs",
context: "module-specifier",
moduleKind: "require",
files: fileMap([file("app/entry.cjs", "root")]),
unanalyzedFiles: {
get: (path) => (path === unavailable.path ? unavailable : undefined),
hasDirectory: () => false,
},
});
expect(result).toMatchObject({
declared_path: "./addon.node",
resolution_status: "unavailable",
candidate_path: "app/addon.node",
limitations: [
"unpacked companion is missing",
expect.stringContaining("inventoried native-addon file"),
],
});
});
});
describe("bare package root file lookup", () => {
it("uses loader-specific bare package roots when only a sibling .js file exists", () => {
const absentDirectory = fileMap([
@@ -499,6 +499,7 @@ const contextualCandidate = (
}
const hasUrlPathSemantics =
context === "module-specifier" && input.moduleKind === "import";
if (hasUrlPathSemantics) declared = htmlUrlText(declared);
const rejected = rejectDeclaration(
declared,
declaresLiteralPercent(input) || hasUrlPathSemantics,
@@ -1400,7 +1401,9 @@ const findFileCandidate = (
candidatePath: path,
limitations: [
...(unanalyzed.limitations ?? []),
`${path} exists in the inventoried artifact container (sha256 ${unanalyzed.sha256 ?? "unknown"}, ${unanalyzed.bytes === null ? "unknown size" : `${String(unanalyzed.bytes)} bytes`}) but is not a JavaScript, JSON, HTML, source map, or native add-on file; its contents were not analyzed.`,
unanalyzed.kind === undefined
? `${path} exists in the inventoried artifact container (sha256 ${unanalyzed.sha256 ?? "unknown"}, ${unanalyzed.bytes === null ? "unknown size" : `${String(unanalyzed.bytes)} bytes`}) but is not a JavaScript, JSON, HTML, source map, or native add-on file; its contents were not analyzed.`
: `${path} is an inventoried ${unanalyzed.kind} file, but its bytes are unavailable (hash status ${unanalyzed.hash_status ?? "unknown"}); resolution cannot be confirmed.`,
],
};
return null;
@@ -261,7 +261,8 @@ export const reconstructJavaScriptArtifact = async (
),
),
statistics: {
relevant_files: files.files.length,
relevant_files:
files.files.length + files.unavailable_relevant_files.length,
nested_asar_containers: files.containers.length,
text_bytes_read: files.text_bytes_read,
invalid_utf8_files: files.invalid_utf8_files,
@@ -25,6 +25,43 @@ import { semanticCoverageResourceLimits } from "../../domain/javascript/javascri
const SHA256 = "a".repeat(64);
it.each([
["plain export", "const answer = 42; export { answer };", 0],
["aliased export", "const answer = 42; export { answer as result };", 0],
["declaration only", "const answer = 42;", 0],
["genuine assignment", "let answer = 0; answer = 1;", 1],
])(
"projects writes only for actual writes: %s",
(_name, source, writeCount) => {
const graph = graphFor(source);
const binding = graph.nodes.find(
({ kind, label }) => kind === "binding" && label === "answer",
);
if (binding === undefined) throw new Error("Expected answer binding");
const writes = graph.relations.filter(
({ relation, target_node_id }) =>
relation === "writes" && target_node_id === binding.node_id,
);
expect(writes).toHaveLength(writeCount);
const exportReference = graph.nodes.find(
({ identity }) => identity.role_key === "reference:0:export:answer",
);
if (source.includes("export {")) expect(exportReference).toBeDefined();
const trace = queryJavaScriptSemanticGraph(graph, {
seed: { kind: "semantic-node", node_id: binding.node_id },
direction: "backward-provenance",
allowed_relations: ["writes"],
});
expect(trace.status).toBe("found");
expect(trace.summary).toMatchObject({
total_seed_matches: 1,
traversed_relations: writeCount,
});
expect(trace.relations).toHaveLength(writeCount);
},
);
it("projects closure and direct interprocedural flow without execution", () => {
const graph = graphFor(`
const outer = 2;
@@ -648,7 +648,7 @@ function* projectDefinitionsAndReferences(
resolution: "resolved",
evidence: inferredSemanticEvidenceAt(context.file, reference.location),
});
else
else if (reference.role === "write")
addRelation(context.state, {
source: expression,
target: binding,
@@ -140,13 +140,21 @@ export const readJavaScriptArtifactFiles = async (
if (leftPriority !== rightPriority) return leftPriority - rightPriority;
return compareUnicodeCodePoints(left.path, right.path);
});
const unanalyzedFiles = createUnanalyzedArtifactLookup(
snapshot,
context.containers,
);
const unavailableRelevantFiles = [...inventory.unavailableFiles]
.sort(compareUnicodeCodePoints)
.flatMap((path) => {
const unavailable = unanalyzedFiles.get(path);
return unavailable === undefined ? [] : [unavailable];
});
assertExpectedFilesWereVisited(expected, files);
return {
files,
unanalyzed_files: createUnanalyzedArtifactLookup(
snapshot,
context.containers,
),
unanalyzed_files: unanalyzedFiles,
unavailable_relevant_files: unavailableRelevantFiles,
containers: context.containers.sort((left, right) =>
compareUnicodeCodePoints(left.path, right.path),
),
@@ -488,6 +496,10 @@ const createUnanalyzedArtifactLookup = (
const isOpaque = (occurrence: ArtifactOccurrence): boolean =>
occurrence.entry_kind === "file" &&
relevantKind(occurrence.logical_path) === undefined;
const isUnavailableRelevant = (occurrence: ArtifactOccurrence): boolean =>
occurrence.entry_kind === "file" &&
occurrence.artifact_id === null &&
relevantKind(occurrence.logical_path) !== undefined;
return {
get(path) {
if (requested.has(path)) return requested.get(path);
@@ -495,8 +507,10 @@ const createUnanalyzedArtifactLookup = (
snapshot.occurrences[
lowerBound(snapshot.occurrences, path, (entry) => entry.logical_path)
];
const kind = relevantKind(path);
const id =
occurrence?.logical_path === path && isOpaque(occurrence)
occurrence?.logical_path === path &&
(isOpaque(occurrence) || isUnavailableRelevant(occurrence))
? occurrence.artifact_id
: undefined;
const candidate =
@@ -507,7 +521,8 @@ const createUnanalyzedArtifactLookup = (
];
const node = candidate?.artifact_id === id ? candidate : undefined;
const file =
occurrence?.logical_path !== path || !isOpaque(occurrence)
occurrence?.logical_path !== path ||
(!isOpaque(occurrence) && !isUnavailableRelevant(occurrence))
? undefined
: {
path,
@@ -515,6 +530,10 @@ const createUnanalyzedArtifactLookup = (
sha256: node?.sha256 ?? null,
bytes: node?.size ?? occurrence.declared_size,
inventory_artifact_id: occurrence.artifact_id,
inventory_occurrence_id: occurrence.occurrence_id,
...(kind === undefined ? {} : { kind }),
artifact_kind: occurrence.artifact_kind,
hash_status: occurrence.hash_status,
limitations: occurrence.limitations,
};
requested.set(path, file);
@@ -539,7 +558,7 @@ const createUnanalyzedArtifactLookup = (
if (entry === undefined || !entry.logical_path.startsWith(prefix))
break;
if (
isOpaque(entry) &&
(isOpaque(entry) || isUnavailableRelevant(entry)) &&
(containerSha256 === undefined ||
containerFor(entry.logical_path) === containerSha256)
) {
+29
View File
@@ -600,6 +600,35 @@ describe("Apple application bridge candidates", () => {
status: "complete",
});
});
it("keeps an extensionless Mach-O without an application root as a native component", () => {
const result = project(
inventoryEvidence("ipa", "Fixture.ipa", [
{ path: "Payload/Fixture.app/Info.plist" },
macho("Payload/Fixture.app/Fixture"),
{ path: "SwiftSupport/outside.js", format: "javascript-bundle" },
macho("SwiftSupport/HelperTool"),
macho("SwiftSupport/iphoneos/libswiftCore.dylib"),
]),
);
expect(result.components.executables.map(({ path }) => path)).toEqual([
"Payload/Fixture.app/Fixture",
"SwiftSupport/HelperTool",
]);
expect(result.bridge_candidates).toEqual([
{
source_path: "SwiftSupport/outside.js",
native_path: "SwiftSupport/HelperTool",
basis: "javascript-and-native-content",
},
{
source_path: "SwiftSupport/outside.js",
native_path: "SwiftSupport/iphoneos/libswiftCore.dylib",
basis: "javascript-and-native-content",
},
]);
});
});
describe("Apple path conventions", () => {
+7 -2
View File
@@ -249,6 +249,9 @@ const appleRootFormat = (format: string): RootFormat => {
return root;
};
const isNativeLibrary = (path: string): boolean =>
/\.(?:dylib|so)$/iu.test(path);
const classifyComponents = (
entries: readonly AppleInventoryEntry[],
all: readonly Component[],
@@ -271,11 +274,13 @@ const classifyComponents = (
bundle_metadata: all.filter(({ path }) =>
anatomyNameIs(lastSegment(path), "Info.plist"),
),
// An unrooted IPA library is listed only under native_libraries.
executables: all.filter(
({ path, format }) => withinApp(path) && isMachOFormat(format),
({ path, format }) =>
isMachOFormat(format) && (withinApp(path) || !isNativeLibrary(path)),
),
frameworks: all.filter(({ path }) => /\.framework\//u.test(path)),
native_libraries: all.filter(({ path }) => /\.(?:dylib|so)$/iu.test(path)),
native_libraries: all.filter(({ path }) => isNativeLibrary(path)),
javascript: all.filter(({ format }) => format === "javascript-bundle"),
signing: all.filter(
({ path }) =>
@@ -140,6 +140,40 @@ it("preserves an observed empty contextBridge API object", () => {
expect(result.parse_status).toBe("complete");
});
it.each([
[
"explicit nested paths after an unknown spread",
`{ ...extra, a: { b: 1 }, c: 2 }`,
["a", "a.b", "c"],
1,
"dynamic",
],
[
"a direct object literal",
`{ a: { b: 1 }, c: 2 }`,
["a", "a.b", "c"],
0,
"object-literal",
],
["a dynamic API argument", "getApi()", [], 1, "dynamic"],
] as const)(
"preserves directly observed nested bridge members for %s",
(_name, api, members, unknownMembers, apiStatus) => {
const result = analyzeJavaScriptStaticSource(`
const { contextBridge } = require("electron");
contextBridge.exposeInMainWorld("api", ${api});
`);
expect(result.electron.context_bridge_apis).toContainEqual(
expect.objectContaining({
api_key: "api",
api_status: apiStatus,
members,
unknown_members: unknownMembers,
}),
);
},
);
describe("wrapped contextBridge exposure methods", () => {
it("retains the original expression for a dynamic isolated world id", () => {
const result = analyzeJavaScriptStaticSource(
@@ -36,6 +36,7 @@ import {
symbolicDefiniteExit,
symbolicLogicalRight,
} from "./javascriptSymbolicControlFlow.js";
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
const callLabel = (node: t.Node): string =>
t.isSequenceExpression(node)
@@ -410,7 +411,12 @@ export const collectElectronWrappedCallPaths = (
api_key_expression: null,
api_status: "dynamic" as const,
resolution_basis: "wrapper-substitution" as const,
members,
// The direct syntax pass recursively retains literal dotted paths. The
// symbolic resolver enumerates only immediate object keys, so its
// replacement must not erase paths already established at this call.
members: [
...new Set([...(existing?.value.members ?? []), ...members]),
].sort(compareUnicodeCodePoints),
unknown_members: resolver.unknownMembers(object),
module_key: null,
location,
@@ -1,4 +1,5 @@
import type { JavaScriptSourceSelection } from "./javascriptSourceSelection.js";
import type { ArtifactOccurrence } from "../artifactGraph.js";
/** Whether an inventoried path has a source extension read by the analyzer. */
export const isJavaScriptSourcePath = (path: string): boolean =>
@@ -34,9 +35,8 @@ export interface JavaScriptArtifactFile {
}
/**
* One inventoried regular file outside the JavaScript-relevant kinds. Its presence
* was inventoried, including unavailable or unhashed members. The source projection
* does not read its bytes; references remain present-but-unanalyzed rather than absent.
* One inventoried regular file not backed by analyzed source bytes. Its inventory
* occurrence remains available even when content is opaque or unavailable.
*/
export interface JavaScriptUnanalyzedArtifactFile {
readonly path: string;
@@ -44,6 +44,14 @@ export interface JavaScriptUnanalyzedArtifactFile {
readonly sha256: string | null;
readonly bytes: number | null;
readonly inventory_artifact_id: string | null;
readonly inventory_occurrence_id?: string;
readonly kind?: JavaScriptArtifactFileKind;
readonly artifact_kind?: ArtifactOccurrence["artifact_kind"];
readonly hash_status?:
| "verified"
| "mismatched"
| "not-hashed"
| "unavailable";
readonly limitations?: readonly string[];
}
@@ -66,6 +74,8 @@ export interface JavaScriptArtifactFileSet {
readonly files: readonly JavaScriptArtifactFile[];
/** Borrowed inventory lookup for regular files the source projection does not read. */
readonly unanalyzed_files?: JavaScriptUnanalyzedArtifactLookup;
/** Relevant inventoried files whose bytes were unavailable. */
readonly unavailable_relevant_files: readonly JavaScriptUnanalyzedArtifactFile[];
readonly containers: readonly JavaScriptArtifactContainer[];
readonly text_bytes_read: number;
readonly invalid_utf8_files: number;
@@ -16,7 +16,7 @@ export const moduleResolutionModeForJavaScriptReference = (
if (referenceKind === "require") return "require";
if (referenceKind === "dynamic-import") return "import";
const extension = posix.extname(sourcePath).toLowerCase();
if (extension === ".cts") return "require";
if (extension === ".cjs" || extension === ".cts") return "require";
if (extension === ".mts") return "import";
if (extension === ".ts" || extension === ".tsx") return "compiler-dependent";
return "import";
@@ -1,6 +1,11 @@
import { describe, expect, it } from "vitest";
import { analyzeJavaScriptSemantics } from "./javascriptSemanticAnalysis.js";
import {
analyzeJavaScriptSemantics,
analyzeParsedJavaScriptSemantics,
} from "./javascriptSemanticAnalysis.js";
import { parseJavaScriptSource } from "./javascriptSourceParser.js";
import { moduleResolutionModeForJavaScriptReference } from "./javascriptModuleResolutionMode.js";
import {
origin,
topLevelBinding,
@@ -56,6 +61,74 @@ describe("TypeScript import-equals links", () => {
});
});
describe("CommonJS wrapper parameter redeclarations", () => {
it("retains ambient exports and require through uninitialized top-level vars", () => {
const path = "app/main.cjs";
const parsed = parseJavaScriptSource(
'var module, exports, require; module.exports = require("./dependency.cjs");',
path,
);
expect(parsed).not.toBeNull();
const ir = analyzeParsedJavaScriptSemantics(
parsed!,
moduleResolutionModeForJavaScriptReference(path, "static-import"),
);
expect(ir.moduleLinks).toEqual(
expect.arrayContaining([
expect.objectContaining({
kind: "commonjs-export",
specifier: "./dependency.cjs",
localName: null,
exportedName: "default",
}),
]),
);
});
it("retains the Node loader call on an uninitialized wrapper var assignment RHS", () => {
const path = "app/main.cjs";
const parsed = parseJavaScriptSource(
'var require; require = require("./bootstrap.cjs");',
path,
);
expect(parsed).not.toBeNull();
const ir = analyzeParsedJavaScriptSemantics(
parsed!,
moduleResolutionModeForJavaScriptReference(path, "static-import"),
);
expect(ir.moduleLinks).toEqual(
expect.arrayContaining([
expect.objectContaining({
kind: "require",
specifier: "./bootstrap.cjs",
}),
]),
);
});
it.each([
'var require = local; require("./dependency.cjs");',
'var require; require = replacement; require("./dependency.cjs");',
"var module; module = replacement; module.exports = 1;",
"var exports; exports = replacement; exports.value = 1;",
'let require; require("./dependency.cjs");',
'function nested() { var require; require("./dependency.cjs"); }',
])("keeps non-wrapper declarations as shadows: %s", (source) => {
const path = "app/main.cjs";
const parsed = parseJavaScriptSource(source, path);
expect(parsed).not.toBeNull();
const ir = analyzeParsedJavaScriptSemantics(
parsed!,
moduleResolutionModeForJavaScriptReference(path, "static-import"),
);
expect(
ir.moduleLinks.filter(
({ kind }) => kind === "require" || kind === "commonjs-export",
),
).toEqual([]);
});
});
describe("TypeScript type-only module links", () => {
it("retains declarations without runtime bindings or callable claims", () => {
const ir = analyzeJavaScriptSemantics(`
@@ -5,6 +5,7 @@ import { BUILTIN_RESOURCE_METHODS } from "./javascriptBuiltinResourceMethods.js"
import type { JavaScriptModuleOrigin } from "./javascriptSemanticIr.js";
import {
resolveSemanticBindingState,
isUnmodifiedCommonJsWrapperBinding,
semanticResolutionBlocked,
semanticStableInitializer,
type JavaScriptSemanticAnalysisState,
@@ -46,13 +47,7 @@ export const semanticRequireLoaderAt = (
return binding.initializers.length === 0 ? "factory-loader" : "unknown";
if (isWrapperRequireInitializerCall(binding, node, state))
return "commonjs-loader";
const scope = state.scopesById.get(binding.scopeId);
if (
scope?.kind === "program" &&
binding.declarationKinds.size > 0 &&
[...binding.declarationKinds].every((kind) => kind === "var") &&
binding.initializers.length === 0
)
if (isUnmodifiedCommonJsWrapperBinding(node.callee, "require", state))
return "commonjs-loader";
if (isSemanticCreateRequireBinding(binding, state)) return "create-require";
const initializer = semanticStableInitializer(binding, state)?.node;
@@ -214,10 +209,9 @@ const writesModuleRequire = (
const value = unwrapJavaScriptExpression(node).node;
if (!t.isIdentifier(value)) return false;
if (value.name === "module")
return (
resolveSemanticBindingState(state, value, "module") === undefined &&
!semanticResolutionBlocked(state, value, "module")
);
return resolveSemanticBindingState(state, value, "module") === undefined
? !semanticResolutionBlocked(state, value, "module")
: isUnmodifiedCommonJsWrapperBinding(value, "module", state);
const binding = resolveSemanticBindingState(state, value, value.name);
if (binding === undefined || visited.has(binding.bindingId)) return false;
const next = new Set(visited).add(binding.bindingId);
@@ -453,10 +447,17 @@ export const semanticRequireOrigin = (
)
)
return undefined;
const requireBinding = resolveSemanticBindingState(
state,
current.callee,
"require",
);
if (
resolveSemanticBindingState(state, current.callee, "require") !==
undefined ||
semanticResolutionBlocked(state, current.callee, "require") ||
(requireBinding !== undefined &&
!isUnmodifiedCommonJsWrapperBinding(current.callee, "require", state) &&
!isWrapperRequireInitializerCall(requireBinding, current, state)) ||
(requireBinding === undefined &&
semanticResolutionBlocked(state, current.callee, "require")) ||
(state.ambientBindingWrites.get("require") ?? []).some(
(write) =>
!semanticTopLevelEffectFollowsCapture(
@@ -253,5 +253,28 @@ export const isUnshadowedGlobal = (
name: string,
): boolean =>
t.isIdentifier(node, { name }) &&
resolveSemanticBindingState(state, node, name) === undefined &&
!semanticResolutionBlocked(state, node, name);
(resolveSemanticBindingState(state, node, name) === undefined
? !semanticResolutionBlocked(state, node, name)
: isUnmodifiedCommonJsWrapperBinding(node, name, state));
/** A program-level `var` redeclaration still denotes its untouched CJS wrapper parameter. */
export const isUnmodifiedCommonJsWrapperBinding = (
node: t.Node,
name: string,
state: JavaScriptSemanticAnalysisState,
): boolean => {
if (
state.staticImportResolutionMode !== "require" ||
!["module", "exports", "require"].includes(name) ||
!t.isIdentifier(node, { name })
)
return false;
const binding = resolveSemanticBindingState(state, node, name);
if (binding === undefined || binding.initializers.length !== 0) return false;
const scope = state.scopesById.get(binding.scopeId);
return (
scope?.kind === "program" &&
binding.declarationKinds.size > 0 &&
[...binding.declarationKinds].every((kind) => kind === "var")
);
};
@@ -62,6 +62,37 @@ it("requires producer coverage accounting in every parsed capture", () => {
);
});
it("does not infer normalized retention budget from missing scenario defaults", () => {
const capture = emptyCapture();
const scenario = {
...capture.manifest.scenario,
limits: {
files: 10_000,
file_bytes: 10_000_000,
processes: 1_000,
filesystem_depth: 16,
},
};
expect(() =>
parseProcessCapture({
...capture,
manifest: {
...capture.manifest,
scenario,
full_scenario_sha256: digestProcessCommitment(scenario),
},
truncation_details: {
...capture.truncation_details,
normalized_terminal: {
budget_bytes: 1_000_000,
retained_bytes: 0,
omitted_frames: 0,
},
},
}),
).toThrow("manifest.scenario.limits.output_bytes");
});
it("rejects settlement and cleanup combinations that cannot occur", () => {
const capture = emptyCapture();
expect(() =>
+17 -1
View File
@@ -8,6 +8,7 @@ const byteRetentionSchema = z.strictObject({
retained_bytes: z.number().int().nonnegative(),
observed_frames: z.number().int().nonnegative(),
retained_frames: z.number().int().nonnegative(),
unknown_size_frames: z.number().int().nonnegative().optional(),
});
/** Filesystem enumeration and whole-file digest coverage for one checkpoint. */
@@ -54,8 +55,18 @@ export const processCaptureTruncationDetailsSchema = z.strictObject({
"UTF-8 bytes of original PTY chunks; whole chunks are retained or omitted.",
),
rendered_terminal: byteRetentionSchema.describe(
"Cumulative UTF-8 bytes of serialized state plus visible lines, evaluated from retained PTY input; whole frames are retained or omitted.",
"Cumulative UTF-8 bytes of serialized state plus visible lines, evaluated from retained PTY input; whole frames are retained or omitted. When normalization exceeds a bound before the frame size is known, observed_bytes contains the exact total for size-known frames and unknown_size_frames counts the other omissions.",
),
normalized_terminal: z
.strictObject({
budget_bytes: z.number().int().nonnegative(),
retained_bytes: z.number().int().nonnegative(),
omitted_frames: z.number().int().nonnegative(),
})
.optional()
.describe(
"UTF-8 bytes of normalized raw-chunk text retained under output_bytes; omitted frames can have unknown final size when normalization exceeded an intermediate bound.",
),
filesystem_before: filesystemCoverageSchema,
filesystem_after: filesystemCoverageSchema,
process: z.strictObject({
@@ -72,6 +83,9 @@ export type ProcessCaptureTruncationDetails = z.infer<
typeof processCaptureTruncationDetailsSchema
>;
export type TerminalRetention = ProcessCaptureTruncationDetails["raw_terminal"];
export type NormalizedTerminalRetention = NonNullable<
ProcessCaptureTruncationDetails["normalized_terminal"]
>;
export type ProcessSamplingCoverage =
ProcessCaptureTruncationDetails["process"];
@@ -88,6 +102,8 @@ export const hasCaptureTruncation = (
): boolean =>
omitted(details.raw_terminal) ||
omitted(details.rendered_terminal) ||
(details.normalized_terminal?.omitted_frames ?? 0) > 0 ||
(details.rendered_terminal.unknown_size_frames ?? 0) > 0 ||
filesystemTruncated(details.filesystem_before) ||
filesystemTruncated(details.filesystem_after) ||
details.process.sampling_partial;
+29 -3
View File
@@ -1,4 +1,7 @@
import { digestProcessCommitment } from "./processScenario.js";
import {
digestProcessCommitment,
processScenarioOutputBudgetProjectionSchema,
} from "./processScenario.js";
import { hasCaptureTruncation } from "./processCaptureCoverage.js";
import type {
@@ -312,8 +315,31 @@ const validateCoverage = (
retention.observed_bytes >= retention.retained_bytes &&
(retention.observed_frames > count ||
retention.observed_bytes === retention.retained_bytes) &&
retention.retained_bytes <=
retention.budget_bytes, `truncation_details.${name}`, "retained observations and budget accounting do not agree");
retention.retained_bytes <= retention.budget_bytes &&
(retention.unknown_size_frames ?? 0) <=
retention.observed_frames -
retention.retained_frames, `truncation_details.${name}`, "retained observations and budget accounting do not agree");
}
const normalized = details.normalized_terminal;
if (normalized !== undefined) {
const committedOutputBudget =
processScenarioOutputBudgetProjectionSchema.safeParse(
capture.manifest.scenario["limits"],
);
require(committedOutputBudget.success, "manifest.scenario.limits.output_bytes", "committed process scenario must include an explicit valid output_bytes budget");
const retainedBytes = capture.frames.reduce(
(total, frame) => total + Buffer.byteLength(frame.data),
0,
);
if (committedOutputBudget.success)
require(normalized.budget_bytes ===
committedOutputBudget.data.output_bytes &&
normalized.retained_bytes === retainedBytes &&
normalized.retained_bytes <= normalized.budget_bytes &&
normalized.omitted_frames <=
details.raw_terminal.observed_frames -
details.raw_terminal
.retained_frames, "truncation_details.normalized_terminal", "normalized terminal retention must agree with the committed budget and retained frames");
}
for (const [name, coverage, files, checkpoint] of [
[
+4
View File
@@ -3,6 +3,10 @@ import canonicalize from "canonicalize";
import { z } from "zod";
const positiveBudget = z.number().int().safe().positive();
/** Read the explicitly committed output budget without applying scenario defaults. */
export const processScenarioOutputBudgetProjectionSchema = z
.object({ output_bytes: positiveBudget })
.passthrough();
/**
* Maximum xterm backing cells for one visible-plus-scrollback buffer. Each
* xterm BufferLine stores three Uint32 words per cell (12 MB at this cap),
+67
View File
@@ -7,6 +7,7 @@ import {
buildManagedPeFixture,
managedPeFixtureTarget,
} from "./ManagedPe.fixture.js";
import { parseExceptionRegions } from "./ManagedMemberInstructionDecoder.js";
const managedBodyWithSections = (...sections: Buffer[]) => {
const header = Buffer.alloc(12);
@@ -666,6 +667,59 @@ describe("managed exception region flags and filters", () => {
});
describe("managed exception section decoding", () => {
it("reads legacy small EH DataSize without counting its header", () => {
const section = Buffer.alloc(16);
section[0] = 0x01;
section[1] = 12;
expect(managedBodyWithSections(section)).toMatchObject({
status: "present",
issue: null,
exception_regions: [expect.any(Object)],
normalized_il_sha256: expect.any(String),
});
});
it("advances a chained legacy section by its physical size", () => {
const legacy = Buffer.alloc(16);
legacy[0] = 0x81;
legacy[1] = 12;
const conformant = Buffer.alloc(16);
conformant[0] = 0x01;
conformant[1] = 16;
expect(managedBodyWithSections(legacy, conformant)).toMatchObject({
status: "present",
issue: null,
exception_regions: [expect.any(Object), expect.any(Object)],
normalized_il_sha256: expect.any(String),
});
});
it("rejects a legacy-sized clause whose physical bytes leave the artifact", () => {
const section = Buffer.alloc(12);
section[0] = 0x01;
section[1] = 12;
expect(parseExceptionRegions(section, 0, section.length)).toMatchObject({
status: "malformed",
regions: [],
issue: "Exception section size leaves artifact",
});
});
it("still rejects a small EH DataSize that matches neither layout", () => {
const section = Buffer.alloc(17);
section[0] = 0x01;
section[1] = 13;
expect(managedBodyWithSections(section)).toMatchObject({
status: "malformed",
exception_regions: [],
issue: "Exception section size does not contain whole clauses",
});
});
it("marks an exception section with a partial clause as malformed", () => {
const section = Buffer.alloc(17);
section[0] = 0x01;
@@ -695,6 +749,19 @@ describe("managed exception section decoding", () => {
});
});
it("keeps small section reserved bytes invalid", () => {
const section = Buffer.alloc(16);
section[0] = 0x01;
section[1] = 16;
section[2] = 1;
expect(managedBodyWithSections(section)).toMatchObject({
status: "malformed",
exception_regions: [],
issue: "Small exception section reserved bytes are nonzero",
});
});
it("preserves decoded clauses when a chained section has an unsupported kind", () => {
const first = Buffer.alloc(16);
first[0] = 0x81;
+27 -3
View File
@@ -543,11 +543,36 @@ export const parseExceptionRegions = (
"Exception section size leaves artifact",
regions,
);
if ((size - 4) % clauseSize !== 0)
let count: number;
let sectionSize = size;
if (fat) {
if ((size - 4) % clauseSize !== 0)
return malformedExceptionRegions(
"Exception section size does not contain whole clauses",
regions,
);
count = (size - 4) / clauseSize;
} else if ((size - 4) % clauseSize === 0) {
count = (size - 4) / clauseSize;
} else if (size >= clauseSize && size % clauseSize === 0) {
// CoreCLR accommodates legacy VB/MC++ sections whose DataSize omits the
// four-byte section header. DataSize is still not the physical extent.
count = size / clauseSize;
sectionSize = 4 + count * clauseSize;
} else {
return malformedExceptionRegions(
"Exception section size does not contain whole clauses",
regions,
);
}
if (
sectionOffset > methodEnd - sectionSize ||
sectionOffset > bytes.length - sectionSize
)
return malformedExceptionRegions(
"Exception section size leaves artifact",
regions,
);
if (
!fat &&
(bytes[sectionOffset + 2] !== 0 || bytes[sectionOffset + 3] !== 0)
@@ -557,7 +582,6 @@ export const parseExceptionRegions = (
regions,
);
const count = (size - 4) / clauseSize;
// Accumulates with the method IL bound; the body reserves what it retains.
budget?.admitTransient(count * 256, sectionOffset);
for (const region of readExceptionClauses(
@@ -569,7 +593,7 @@ export const parseExceptionRegions = (
regions.push(region);
hasMoreSections = (kind & 0x80) !== 0;
sectionOffset += size;
sectionOffset += sectionSize;
}
return { status: "complete", regions };
};
+71 -1
View File
@@ -1,6 +1,14 @@
import { createHash } from "node:crypto";
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { hopperLoaderArgsForTarget } from "./HopperAnalysisProfile.js";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import {
hopperLoaderArgsForTarget,
resolveHopperAnalysisProfile,
} from "./HopperAnalysisProfile.js";
it("routes physical universal Mach-O containers through Hopper's FAT loader", () => {
const result = hopperLoaderArgsForTarget(
@@ -21,3 +29,65 @@ it("routes physical universal Mach-O containers through Hopper's FAT loader", ()
value: ["-l", "FAT", "--aarch64", "-l", "Mach-O"],
});
});
it("prepares a verified thin arm64e slice for a FAT32 system container", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-profile-");
const writeUInt32 = (buffer: Buffer, offset: number, value: number): void => {
buffer.writeUInt32BE(value, offset);
};
const slices = [
{ cpuType: 0x01000007, cpuSubtype: 3 },
{ cpuType: 0x0100000c, cpuSubtype: 0x80000002 },
{ cpuType: 0x0100000c, cpuSubtype: 0x8000000c },
];
const container = Buffer.alloc(3 * 4096 + 64);
writeUInt32(container, 0, 0xcafebabe);
writeUInt32(container, 4, slices.length);
slices.forEach((slice, index) => {
const offset = 4096 * (index + 1);
const entry = 8 + index * 20;
writeUInt32(container, entry, slice.cpuType);
writeUInt32(container, entry + 4, slice.cpuSubtype);
writeUInt32(container, entry + 8, offset);
writeUInt32(container, entry + 12, 64);
writeUInt32(container, entry + 16, 12);
writeUInt32(container, offset, 0xfeedfacf);
writeUInt32(container, offset + 4, slice.cpuType);
writeUInt32(container, offset + 8, slice.cpuSubtype);
});
const arm64eOffset = 8192;
const path = join(directory, "system.fat");
await writeFile(path, container);
const result = await resolveHopperAnalysisProfile(
{
path,
sha256: createHash("sha256").update(container).digest("hex"),
kind: "executable",
format: "mach-o-universal",
architecture: "arm64",
availableArchitectures: ["x86_64", "arm64"],
universalSlice: { offset: arm64eOffset, size: 64 },
},
{
launcherPath: path,
loaderArgsOverride: [],
provider: { id: "hopper", name: "Hopper Disassembler", version: null },
},
);
if (!result.ok) throw new Error(result.error.message);
const parameters = result.value.profile?.parameters;
expect(parameters).toMatchObject({
macho_container: "fat32",
prepared_image: {
method: "fat32_thin_slice",
source_offset: arm64eOffset,
byte_length: 64,
cpu_type: 0x0100000c,
cpu_subtype: 0x80000002,
},
loader: {
source: "derived",
arguments: ["-l", "Mach-O", "--aarch64"],
},
});
});
+1 -1
View File
@@ -54,7 +54,7 @@ export const resolveHopperAnalysisProfile = async (
let preparedImage: HopperMachOImage | undefined;
if (
target.kind === "executable" &&
machoContainer === "fat64" &&
(machoContainer === "fat64" || machoContainer === "fat32") &&
options.loaderArgsOverride.length === 0
) {
const resolved = await resolveHopperMachOImage(target, options.signal);
+331 -4
View File
@@ -1,11 +1,109 @@
import { chmod, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { chmod, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { resolveHopperMachOImage } from "./HopperMachOImage.js";
import {
prepareHopperMachOImage,
resolveHopperMachOImage,
} from "./HopperMachOImage.js";
const CPU_TYPES = { x86_64: 0x01000007, arm64: 0x0100000c } as const;
const FAT32_ALIGNMENT = 12;
const FAT32_SLICE_SIZE = 64;
const ARM64E = 2;
const ARM64E_X1 = 12;
const ARM64E_WITH_CAPABILITIES = 0x80000002;
const ARM64E_X1_WITH_CAPABILITIES = 0x8000000c;
interface FatFixtureSlice {
readonly cpuType: number;
readonly cpuSubtype: number;
/** Thin-slice byte order, independent of the container's table byte order. */
readonly little?: boolean;
}
interface Fat32Fixture {
readonly container: Buffer;
readonly sliceOffsets: readonly number[];
}
/** Synthesize a FAT32 container whose slices are 4096-aligned thin headers. */
const buildFat32Container = (
slices: readonly FatFixtureSlice[],
options: { readonly littleEndianTable?: boolean } = {},
): Fat32Fixture => {
const littleTable = options.littleEndianTable === true;
const writeTableUInt32 = littleTable
? (buffer: Buffer, offset: number, value: number): number =>
buffer.writeUInt32LE(value, offset)
: (buffer: Buffer, offset: number, value: number): number =>
buffer.writeUInt32BE(value, offset);
const unit = 2 ** FAT32_ALIGNMENT;
const sliceOffsets = slices.map((_slice, index) => unit * (index + 1));
const lastOffset = sliceOffsets[sliceOffsets.length - 1] ?? unit;
const container = Buffer.alloc(lastOffset + FAT32_SLICE_SIZE, 0);
// The magic VALUE is 0xcafebabe in the table's byte order: little-endian
// storage yields the FAT_CIGAM magic a big-endian reader observes.
writeTableUInt32(container, 0, 0xcafebabe);
writeTableUInt32(container, 4, slices.length);
for (const [index, slice] of slices.entries()) {
const entry = 8 + index * 20;
const offset = sliceOffsets[index]!;
writeTableUInt32(container, entry, slice.cpuType);
writeTableUInt32(container, entry + 4, slice.cpuSubtype);
writeTableUInt32(container, entry + 8, offset);
writeTableUInt32(container, entry + 12, FAT32_SLICE_SIZE);
writeTableUInt32(container, entry + 16, FAT32_ALIGNMENT);
const writeThinUInt32 =
slice.little === true
? (buffer: Buffer, fieldOffset: number, value: number): number =>
buffer.writeUInt32LE(value, fieldOffset)
: (buffer: Buffer, fieldOffset: number, value: number): number =>
buffer.writeUInt32BE(value, fieldOffset);
writeThinUInt32(container, offset, 0xfeedfacf);
writeThinUInt32(container, offset + 4, slice.cpuType);
writeThinUInt32(container, offset + 8, slice.cpuSubtype);
}
return { container, sliceOffsets };
};
const writeFixture = async (
directory: string,
name: string,
fixture: Fat32Fixture,
): Promise<string> => {
const path = join(directory, name);
await writeFile(path, fixture.container);
return path;
};
const fat32Target = (
path: string,
fixture: Fat32Fixture,
architecture: "arm64" | "x86_64",
) => ({
path,
sha256: createHash("sha256").update(fixture.container).digest("hex"),
kind: "executable" as const,
format: "mach-o-universal" as const,
architecture,
availableArchitectures: ["x86_64", "arm64"] as const,
universalSlice: { offset: fixture.sliceOffsets[0]!, size: FAT32_SLICE_SIZE },
});
const requireResolved = (
result: Awaited<ReturnType<typeof resolveHopperMachOImage>>,
) => {
if (!result.ok)
throw new Error(
`FAT fixture resolution failed: ${JSON.stringify(projectAnalysisError(result.error))}`,
);
return result.value;
};
it.skipIf(process.platform === "win32" || process.getuid?.() === 0)(
"reports a FAT64 source that became unreadable as a host read denial",
@@ -36,7 +134,7 @@ it.skipIf(process.platform === "win32" || process.getuid?.() === 0)(
},
);
it("rejects an ELF target before attempting FAT64 source preparation", async () => {
it("rejects an ELF target before attempting FAT source preparation", async () => {
const directory = await createTestTempDirectory("rea-hopper-mips-boundary-");
// No file is created: format validation must precede any filesystem lookup.
const path = join(directory, "mips.elf");
@@ -54,6 +152,235 @@ it("rejects an ELF target before attempting FAT64 source preparation", async ()
expect(result.error).toMatchObject({
_tag: "BinaryTargetError",
path,
reason: "Hopper FAT64 image preparation requires a Mach-O target",
reason: "Hopper FAT image preparation requires a Mach-O target",
});
});
it("selects the plain-arm64-decodable arm64e slice from an Apple-style FAT32 container", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-arm64e-");
// /bin/* system binaries: x86_64, arm64e (cpusubtype 2), arm64e.x1 (subtype 12).
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
]);
const path = await writeFixture(directory, "system.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(result).toMatchObject({
method: "fat32_thin_slice",
architecture_count: 3,
source_offset: fixture.sliceOffsets[1],
byte_length: FAT32_SLICE_SIZE,
cpu_type: CPU_TYPES.arm64,
cpu_subtype: ARM64E,
});
});
it("ranks capability-bearing arm64 subtypes by base and retains raw identity", async () => {
const directory = await createTestTempDirectory(
"rea-hopper-fat32-capabilities-",
);
const fixture = buildFat32Container([
{
cpuType: CPU_TYPES.arm64,
cpuSubtype: ARM64E_X1_WITH_CAPABILITIES,
},
{
cpuType: CPU_TYPES.arm64,
cpuSubtype: ARM64E_WITH_CAPABILITIES,
},
]);
const path = await writeFixture(directory, "system.fat", fixture);
const selected = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(selected).toMatchObject({
source_offset: fixture.sliceOffsets[1],
cpu_subtype: ARM64E_WITH_CAPABILITIES,
});
});
it("fails closed when capability variants share the preferred arm64e base", async () => {
const directory = await createTestTempDirectory(
"rea-hopper-fat32-capability-ambiguity-",
);
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: 0x80000002 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: 0x81000002 },
]);
const path = await writeFixture(directory, "ambiguous.fat", fixture);
const result = await resolveHopperMachOImage(
fat32Target(path, fixture, "arm64"),
);
if (result.ok || result.error._tag !== "BinaryTargetError")
throw new Error(
"Expected ambiguous FAT selection to return a BinaryTargetError",
);
expect(result.error.reason).toContain("selection for arm64 is ambiguous");
});
it("prefers cpusubtype 2 over 12 regardless of FAT32 table order", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-order-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
]);
const path = await writeFixture(directory, "reordered.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(result).toMatchObject({
cpu_subtype: ARM64E,
source_offset: fixture.sliceOffsets[1],
});
});
it("prefers a plain arm64 slice over arm64e in a FAT32 container", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-plain-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: 0 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
]);
const path = await writeFixture(directory, "plain.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(result).toMatchObject({
cpu_subtype: 0,
source_offset: fixture.sliceOffsets[0],
});
});
it("keeps a lone arm64e.x1 FAT32 slice selectable to preserve today's open behavior", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-lone-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
]);
const path = await writeFixture(directory, "lone.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(result).toMatchObject({ cpu_subtype: ARM64E_X1 });
});
it("still fails closed when two arm64e.x1 FAT32 slices compete", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-ambig-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
]);
const path = await writeFixture(directory, "ambiguous.fat", fixture);
const result = await resolveHopperMachOImage(
fat32Target(path, fixture, "arm64"),
);
if (result.ok) throw new Error("Ambiguous FAT32 selection succeeded");
expect(
projectAnalysisError(result.error).details?.reason ?? result.error,
).toMatch(/selection.*ambiguous/u);
});
it("selects the single x86_64 slice of a FAT32 container for an x86_64 target", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-intel-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
{ cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 },
]);
const path = await writeFixture(directory, "universal.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "x86_64")),
);
expect(result).toMatchObject({
cpu_type: CPU_TYPES.x86_64,
cpu_subtype: 3,
source_offset: fixture.sliceOffsets[1],
});
});
it("parses a little-endian FAT32 table without tying it to slice byte order", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-le-");
const fixture = buildFat32Container(
[
{ cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
],
{ littleEndianTable: true },
);
const path = await writeFixture(directory, "little.fat", fixture);
const result = requireResolved(
await resolveHopperMachOImage(fat32Target(path, fixture, "arm64")),
);
expect(result).toMatchObject({
method: "fat32_thin_slice",
cpu_subtype: ARM64E,
source_offset: fixture.sliceOffsets[1],
});
});
it("rejects a FAT32 slice whose declared range extends beyond the file", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-range-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
]);
fixture.container.writeUInt32BE(fixture.container.length + 16, 8 + 12);
const path = await writeFixture(directory, "overrange.fat", fixture);
const result = await resolveHopperMachOImage(
fat32Target(path, fixture, "arm64"),
);
if (result.ok) throw new Error("An overranging FAT32 slice was accepted");
expect(
projectAnalysisError(result.error).details?.reason ?? result.error,
).toMatch(/extends beyond the file/u);
});
it("copies exactly the selected FAT32 slice bytes and verifies the source", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-copy-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.x86_64, cpuSubtype: 3 },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
]);
const path = await writeFixture(directory, "system.fat", fixture);
const target = fat32Target(path, fixture, "arm64");
const resolved = requireResolved(await resolveHopperMachOImage(target));
const destination = join(directory, "image.macho");
const prepared = await prepareHopperMachOImage(
path,
target.sha256,
resolved,
destination,
);
if (!prepared.ok) throw new Error(prepared.error.userMessage);
const arm64eOffset = fixture.sliceOffsets[1]!;
const copied = fixture.container.subarray(
arm64eOffset,
arm64eOffset + FAT32_SLICE_SIZE,
);
expect(await readFile(destination)).toEqual(copied);
});
it("rejects FAT32 preparation when the source table changed after resolution", async () => {
const directory = await createTestTempDirectory("rea-hopper-fat32-race-");
const fixture = buildFat32Container([
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E },
{ cpuType: CPU_TYPES.arm64, cpuSubtype: ARM64E_X1 },
]);
const path = await writeFixture(directory, "system.fat", fixture);
const target = fat32Target(path, fixture, "arm64");
const resolved = requireResolved(await resolveHopperMachOImage(target));
const mutated = Buffer.from(fixture.container);
mutated.writeUInt32BE(0x01000007, 8);
await writeFile(path, mutated);
const destination = join(directory, "image.macho");
const prepared = await prepareHopperMachOImage(
path,
target.sha256,
resolved,
destination,
);
if (prepared.ok)
throw new Error("A mutated FAT32 source passed verification");
expect(prepared.error.userMessage).toMatch(/no longer agrees/u);
});
+66 -30
View File
@@ -8,7 +8,10 @@ import { AnalysisCancelledError } from "../domain/analysisErrorCore.js";
import {
readFatSliceDeclarations,
validateFatSlice,
type FatSliceDeclaration,
} from "../domain/apple/machoContainer.js";
import { machoCpuSubtypeBase } from "../domain/apple/machoCpuIdentity.js";
import type { HeaderArchitecture } from "../domain/binaryTargetTypes.js";
import {
HopperCancelledError,
HopperStartError,
@@ -17,7 +20,10 @@ import { err, ok, type Result } from "../domain/result.js";
/** Stable source coordinates committed before preparing an owned thin image. */
export const hopperMachOImageSchema = z.strictObject({
method: z.literal("fat64_thin_slice"),
method: z.union([
z.literal("fat32_thin_slice"),
z.literal("fat64_thin_slice"),
]),
architecture_count: z.number().int().min(1).max(128),
container_table_sha256: z.string().regex(/^[a-f0-9]{64}$/u),
source_offset: z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER),
@@ -28,7 +34,7 @@ export const hopperMachOImageSchema = z.strictObject({
export type HopperMachOImage = z.infer<typeof hopperMachOImageSchema>;
/** Validate the selected FAT64 slice without depending on Apple's lipo tool. */
/** Validate the selected FAT slice without depending on Apple's lipo tool. */
export const resolveHopperMachOImage = async (
target: BinaryTarget & { readonly kind: "executable" },
signal?: AbortSignal,
@@ -45,14 +51,14 @@ export const resolveHopperMachOImage = async (
);
if (signal?.aborted) return err(new AnalysisCancelledError("open_binary"));
if (target.format !== "mach-o" && target.format !== "mach-o-universal")
return invalid("Hopper FAT64 image preparation requires a Mach-O target");
return invalid("Hopper FAT image preparation requires a Mach-O target");
let file: FileHandle;
try {
file = await open(target.path, "r");
} catch (cause: unknown) {
// Keep the OS error so a read-permission denial reports access_denied.
return invalid(
`Cannot read FAT64 source for Hopper image preparation: ${cause instanceof Error ? cause.message : String(cause)}`,
`Cannot read FAT source for Hopper image preparation: ${cause instanceof Error ? cause.message : String(cause)}`,
cause,
);
}
@@ -64,24 +70,29 @@ export const resolveHopperMachOImage = async (
};
const header = await read(0, 8);
const magic = header.length === 8 ? header.readUInt32BE(0) : 0;
if (magic !== 0xcafebabf && magic !== 0xbfbafeca)
const isFatMagic = [
0xcafebabe, 0xcafebabf, 0xbebafeca, 0xbfbafeca,
].includes(magic);
if (!isFatMagic)
return invalid(
"FAT64 source header changed before Hopper image preparation",
"FAT source header changed before Hopper image preparation",
);
const littleEndian = magic === 0xbfbafeca;
const wide = magic === 0xcafebabf || magic === 0xbfbafeca;
const littleEndian = magic === 0xbebafeca || magic === 0xbfbafeca;
const entrySize = wide ? 32 : 20;
const count = littleEndian
? header.readUInt32LE(4)
: header.readUInt32BE(4);
const size = (await file.stat()).size;
const tableEnd = 8 + count * 32;
const tableEnd = 8 + count * entrySize;
if (count === 0 || count > 128 || tableEnd > size)
return invalid(
"FAT64 architecture table is truncated or exceeds the supported 128 entries",
"FAT architecture table is truncated or exceeds the supported 128 entries",
);
const table = await read(8, count * 32);
const table = await read(8, count * entrySize);
const parsed = readFatSliceDeclarations(table, {
count,
wide: true,
wide,
littleEndian,
});
if (parsed.status === "malformed") return invalid(parsed.reason);
@@ -93,24 +104,26 @@ export const resolveHopperMachOImage = async (
return err(new AnalysisCancelledError("open_binary"));
const rangeError = validateFatSlice(slice, size, tableEnd);
if (rangeError !== null) return invalid(rangeError);
const reserved = littleEndian
? table.readUInt32LE(slice.index * 32 + 28)
: table.readUInt32BE(slice.index * 32 + 28);
if (reserved !== 0)
return invalid(
`FAT64 architecture ${slice.index} has a nonzero reserved field`,
);
if (wide) {
const reserved = littleEndian
? table.readUInt32LE(slice.index * 32 + 28)
: table.readUInt32BE(slice.index * 32 + 28);
if (reserved !== 0)
return invalid(
`FAT64 architecture ${slice.index} has a nonzero reserved field`,
);
}
if (slice.size < 12)
return invalid(
`FAT64 architecture ${slice.index} has a truncated Mach-O header`,
`FAT architecture ${slice.index} has a truncated Mach-O header`,
);
const embedded = await read(slice.offset, 12);
if (embedded.length !== 12)
return invalid(`FAT64 architecture ${slice.index} became truncated`);
return invalid(`FAT architecture ${slice.index} became truncated`);
const thinMagic = embedded.readUInt32BE(0);
if (![0xfeedface, 0xfeedfacf, 0xcefaedfe, 0xcffaedfe].includes(thinMagic))
return invalid(
`FAT64 architecture ${slice.index} does not contain a Mach-O image`,
`FAT architecture ${slice.index} does not contain a Mach-O image`,
);
const little = thinMagic === 0xcefaedfe || thinMagic === 0xcffaedfe;
const identity = {
@@ -133,15 +146,15 @@ export const resolveHopperMachOImage = async (
current !== undefined &&
current.offset < previous.offset + previous.size
)
return invalid("FAT64 slices overlap");
return invalid("FAT slices overlap");
}
const selected = matches.length === 1 ? matches[0] : undefined;
const selected = selectHopperSlice(matches, target.architecture);
if (selected === undefined)
return invalid(
`FAT64 selection for ${target.architecture} is ${matches.length === 0 ? "missing" : "ambiguous"}; extract the exact architecture into a thin image before opening it`,
`FAT selection for ${target.architecture} is ${matches.length === 0 ? "missing" : "ambiguous"}; extract the exact architecture into a thin image before opening it`,
);
return ok({
method: "fat64_thin_slice",
method: wide ? "fat64_thin_slice" : "fat32_thin_slice",
architecture_count: count,
container_table_sha256: createHash("sha256")
.update(header)
@@ -154,13 +167,35 @@ export const resolveHopperMachOImage = async (
});
} catch (cause: unknown) {
return invalid(
`Cannot validate FAT64 image: ${cause instanceof Error ? cause.message : String(cause)}`,
`Cannot validate FAT image: ${cause instanceof Error ? cause.message : String(cause)}`,
);
} finally {
await file.close();
}
};
/**
* Plain arm64 (cpusubtype 0) and arm64e (cpusubtype 2) function-entry encodings
* decode under Hopper's --aarch64; Apple's arm64e.x1 variant (cpusubtype 12)
* does not, so it is selected only when it is the arm64 family's only candidate.
* Every other ambiguity stays fail-closed.
*/
const selectHopperSlice = (
matches: readonly FatSliceDeclaration[],
architecture: HeaderArchitecture,
): FatSliceDeclaration | undefined => {
if (matches.length === 1) return matches[0];
if (architecture !== "arm64") return undefined;
for (const cpuSubtype of [0, 2]) {
const preferred = matches.filter(
(slice) => machoCpuSubtypeBase(slice.cpuSubtype) === cpuSubtype,
);
if (preferred.length === 1) return preferred[0];
if (preferred.length > 1) return undefined;
}
return undefined;
};
/** Copy a selected slice and verify the entire original source in the same pass. */
export const prepareHopperMachOImage = async (
sourcePath: string,
@@ -170,6 +205,7 @@ export const prepareHopperMachOImage = async (
signal?: AbortSignal,
): Promise<Result<string, HopperStartError | HopperCancelledError>> => {
if (signal?.aborted) return err(new HopperCancelledError());
const tableEntrySize = image.method === "fat64_thin_slice" ? 32 : 20;
try {
const source = await open(sourcePath, "r");
try {
@@ -177,7 +213,7 @@ export const prepareHopperMachOImage = async (
try {
const hash = createHash("sha256");
const tableHash = createHash("sha256");
const tableEnd = 8 + image.architecture_count * 32;
const tableEnd = 8 + image.architecture_count * tableEntrySize;
const thinHeader: Buffer[] = [];
let position = 0;
let copied = 0;
@@ -223,13 +259,13 @@ export const prepareHopperMachOImage = async (
)
return err(
new HopperStartError({
userMessage: `FAT64 source ${sourcePath} no longer agrees with its committed architecture table or selected Mach-O header. Open the stable source again.`,
userMessage: `FAT source ${sourcePath} no longer agrees with its committed architecture table or selected Mach-O header. Open the stable source again.`,
}),
);
if (observed !== sourceSha256 || copied !== image.byte_length)
return err(
new HopperStartError({
userMessage: `FAT64 source ${sourcePath} changed before launch: expected SHA-256 ${sourceSha256}, observed ${observed}; copied ${copied} of ${image.byte_length} selected bytes. Open the stable source again.`,
userMessage: `FAT source ${sourcePath} changed before launch: expected SHA-256 ${sourceSha256}, observed ${observed}; copied ${copied} of ${image.byte_length} selected bytes. Open the stable source again.`,
}),
);
return ok(destination);
@@ -245,7 +281,7 @@ export const prepareHopperMachOImage = async (
: err(
new HopperStartError({
cause,
userMessage: `Cannot prepare the selected FAT64 image from ${sourcePath}: ${cause instanceof Error ? cause.message : String(cause)}`,
userMessage: `Cannot prepare the selected FAT image from ${sourcePath}: ${cause instanceof Error ? cause.message : String(cause)}`,
}),
);
}
+1 -1
View File
@@ -288,7 +288,7 @@ export class HopperProvider implements AnalysisProviderCandidate {
...(preparedImage === undefined
? []
: [
"REA loaded a verified thin Mach-O slice from the original FAT64 source. Identity and file offsets refer to the original file. The owned temporary image and its Hopper document close together, including on MCP shutdown.",
"REA loaded a verified thin Mach-O slice from the original FAT source. Identity and file offsets refer to the original file. The owned temporary image and its Hopper document close together, including on MCP shutdown.",
]),
...(CAPABILITIES.find(
(descriptor) => descriptor.operation === operation,
+44 -15
View File
@@ -38,11 +38,13 @@ import { assertNotCancelled } from "./ProcessScenarioRuntimeValidation.js";
import {
normalizeProcessElapsedTime,
normalizeProcessSamples,
type BoundedProcessTextNormalization,
} from "./ProcessNormalization.js";
import { PROCESS_PROVIDER } from "../../domain/process/processEvidenceProvider.js";
import { TerminalRenderer } from "./TerminalRenderer.js";
import {
hasCaptureTruncation,
type NormalizedTerminalRetention,
type TerminalRetention,
type ProcessCaptureTruncationDetails,
} from "../../domain/process/processCaptureCoverage.js";
@@ -408,7 +410,9 @@ const captureCoverageUnknowns = (
unknowns.push({
scope: "terminal",
reason:
"Raw PTY chunks exceeded output_bytes and were omitted; rendered states also lack those inputs. See truncation_details.raw_terminal.",
(details.normalized_terminal?.omitted_frames ?? 0) > 0
? "Raw PTY chunks were omitted because original input exceeded output_bytes or an ordered normalization stage exceeded its remaining bound; final normalized sizes for the latter omissions are unknown. Rendered states also lack those inputs. See raw_terminal and normalized_terminal retention details, including the normalized omission count."
: "Raw PTY chunks exceeded output_bytes and were omitted; rendered states also lack those inputs. See truncation_details.raw_terminal.",
});
if (
details.rendered_terminal.observed_frames >
@@ -417,7 +421,9 @@ const captureCoverageUnknowns = (
unknowns.push({
scope: "terminal",
reason:
"Cumulative rendered state and visible-line bytes exceeded output_bytes; whole rendered frames were omitted. Raw PTY coverage is reported separately. See truncation_details.rendered_terminal.",
(details.rendered_terminal.unknown_size_frames ?? 0) > 0
? "Rendered frames were omitted because a normalization stage exceeded the remaining output_bytes budget; final rendered byte totals are unknown for those frames. See truncation_details.rendered_terminal."
: "Cumulative rendered state and visible-line bytes exceeded output_bytes; whole rendered frames were omitted. Raw PTY coverage is reported separately. See truncation_details.rendered_terminal.",
});
for (const [name, coverage] of [
["before", details.filesystem_before],
@@ -1303,45 +1309,68 @@ export const captureTerminalFrames = (options: {
readonly started: number;
readonly onOutput: () => void;
readonly renderer: TerminalRenderer;
readonly normalizeText: (value: string) => string;
readonly normalizeText: (
value: string,
maxBytes: number,
) => BoundedProcessTextNormalization;
readonly recordEvent: RecordProcessCaptureEvent;
}): (() => TerminalRetention) => {
let outputBytes = 0;
}): (() => TerminalCaptureRetention) => {
let rawBytes = 0;
let normalizedBytes = 0;
let observedBytes = 0;
let observedFrames = 0;
let normalizedOmittedFrames = 0;
options.terminal.onData((data) => {
options.onOutput();
const bytes = Buffer.byteLength(data);
observedBytes += bytes;
observedFrames += 1;
if (outputBytes + bytes > options.scenario.limits.output_bytes) {
const budget = options.scenario.limits.output_bytes;
if (bytes > budget - rawBytes) {
return;
}
outputBytes += bytes;
const normalized = options.normalizeText(data, budget - normalizedBytes);
if (normalized.state === "over_budget") {
normalizedOmittedFrames += 1;
return;
}
rawBytes += bytes;
normalizedBytes += normalized.utf8Bytes;
const atMs = normalizeProcessElapsedTime(
monotonicProcessCaptureClock() - options.started,
options.scenario.normalization.time_bucket_ms,
);
const normalized = options.normalizeText(data);
const sequence = options.frames.length;
options.frames.push({
sequence,
at_ms: atMs,
data: normalized,
...(normalized === data ? {} : { raw_data: data }),
data: normalized.value,
...(normalized.value === data ? {} : { raw_data: data }),
});
options.renderer.write(data, atMs);
options.recordEvent("frames", sequence);
});
return () => ({
budget_bytes: options.scenario.limits.output_bytes,
observed_bytes: observedBytes,
retained_bytes: outputBytes,
observed_frames: observedFrames,
retained_frames: options.frames.length,
raw_terminal: {
budget_bytes: options.scenario.limits.output_bytes,
observed_bytes: observedBytes,
retained_bytes: rawBytes,
observed_frames: observedFrames,
retained_frames: options.frames.length,
},
normalized_terminal: {
budget_bytes: options.scenario.limits.output_bytes,
retained_bytes: normalizedBytes,
omitted_frames: normalizedOmittedFrames,
},
});
};
export interface TerminalCaptureRetention {
readonly raw_terminal: TerminalRetention;
readonly normalized_terminal: NormalizedTerminalRetention;
}
export const resolveProcessResult = (
capture: PendingProcessCapture | ProcessCapture | undefined,
executionFailure: unknown,
@@ -66,7 +66,12 @@ it("returns detached terminal observations", async () => {
rows: 12,
scrollback: 100,
maxBytes: 100_000,
normalize: (value) => value,
normalize: (value, maxBytes) => {
const utf8Bytes = Buffer.byteLength(value);
return utf8Bytes <= maxBytes
? { state: "normalized", value, utf8Bytes }
: { state: "over_budget" };
},
recordEvent: (collection, index) =>
observed.push(`${collection}:${String(index)}`),
});
@@ -0,0 +1,63 @@
import { expect, it } from "vitest";
import type { IPty } from "@lydell/node-pty";
import { parseProcessScenario } from "../../domain/process/processScenario.js";
import type { TerminalFrame } from "../../domain/process/processCapture.js";
import { captureTerminalFrames } from "./ProcessCaptureLifecycle.js";
import { createBoundedProcessTextNormalizer } from "./ProcessNormalization.js";
it("omits an expanding chunk whole and retains a later smaller normalized chunk", () => {
let onData: ((data: string) => void) | undefined;
const terminal = {
onData: (listener: (data: string) => void) => {
onData = listener;
return { dispose: () => undefined };
},
} as unknown as IPty;
const scenario = parseProcessScenario({
executable: "unused",
limits: { output_bytes: 1_000 },
normalization: {
paths: false,
pids: false,
ports: false,
patterns: [{ pattern: "a", replacement: "x".repeat(500) }],
},
});
const frames: TerminalFrame[] = [];
const renderedChunks: string[] = [];
const journal: Array<[string, number]> = [];
const captureRetention = captureTerminalFrames({
terminal,
scenario,
frames,
started: 0,
onOutput: () => undefined,
renderer: { write: (data: string) => renderedChunks.push(data) } as never,
normalizeText: createBoundedProcessTextNormalizer(scenario, "/tmp", -1),
recordEvent: (collection, index) => journal.push([collection, index]),
});
onData?.("aaa");
onData?.("z");
expect(frames).toEqual([
{ sequence: 0, at_ms: expect.any(Number), data: "z" },
]);
expect(renderedChunks).toEqual(["z"]);
expect(journal).toEqual([["frames", 0]]);
expect(captureRetention()).toEqual({
raw_terminal: {
budget_bytes: 1_000,
observed_bytes: 4,
retained_bytes: 1,
observed_frames: 2,
retained_frames: 1,
},
normalized_terminal: {
budget_bytes: 1_000,
retained_bytes: 1,
omitted_frames: 1,
},
});
});
+14 -8
View File
@@ -1,5 +1,4 @@
import type { IPty } from "@lydell/node-pty";
import type { TerminalRetention } from "../../domain/process/processCaptureCoverage.js";
import type { ProcessCapture } from "../../domain/process/processCaptureParsing.js";
import type { ProcessScenario } from "../../domain/process/processScenario.js";
import { parseProcessCapture } from "../../domain/process/processCaptureParsing.js";
@@ -24,7 +23,10 @@ import { startProcessSampler } from "./ProcessSampling.js";
import { snapshotRoots } from "./FilesystemSnapshot.js";
import { TerminalRenderer } from "./TerminalRenderer.js";
import type { ProcessTimer } from "./ProcessTimer.js";
import { createProcessTextNormalizer } from "./ProcessNormalization.js";
import {
createBoundedProcessTextNormalizer,
type BoundedProcessTextNormalization,
} from "./ProcessNormalization.js";
import { selectCapturedProcessGroupIds } from "../ProcessOwnershipProcessTree.js";
import {
awaitTerminalExit,
@@ -46,6 +48,7 @@ import {
describeUnverifiedCause,
type ProcessCaptureResourceCleanupOptions,
ProcessCaptureResourceScope,
type TerminalCaptureRetention,
} from "./ProcessCaptureLifecycle.js";
import {
assertNotCancelled,
@@ -79,7 +82,7 @@ interface StartedCaptureRuntime {
readonly startedAt: Date;
readonly executableIdentity: ReturnType<typeof observeLaunchedExecutable>;
readonly lastOutput: () => number;
readonly rawTerminalRetention: () => TerminalRetention;
readonly terminalRetention: () => TerminalCaptureRetention;
readonly stopSampler: ReturnType<typeof startProcessSampler>;
/** Signals the captured root only while its launch-time start identity still matches. */
readonly signalRoot: (signal: "SIGTERM" | "SIGKILL") => Promise<{
@@ -222,7 +225,10 @@ const cleanupFailedStartup = async (options: {
const createTerminalRenderer = (
scenario: ProcessScenario,
normalizeText: (value: string) => string,
normalizeText: (
value: string,
maxBytes: number,
) => BoundedProcessTextNormalization,
recordEvent: RecordProcessCaptureEvent,
): TerminalRenderer =>
new TerminalRenderer({
@@ -269,7 +275,7 @@ const startCaptureRuntime = async (
const startedAt = new Date();
const started = monotonicProcessCaptureClock();
let lastOutput = started;
const normalizeText = createProcessTextNormalizer(
const normalizeText = createBoundedProcessTextNormalizer(
scenario,
options.temporaryRoot,
() => terminal?.pid ?? -1,
@@ -295,7 +301,7 @@ const startCaptureRuntime = async (
scenario.executable,
selectedExecutable,
);
const rawTerminalRetention = captureTerminalFrames({
const terminalRetention = captureTerminalFrames({
...options,
terminal,
started,
@@ -324,7 +330,7 @@ const startCaptureRuntime = async (
startedAt,
executableIdentity,
lastOutput: () => lastOutput,
rawTerminalRetention,
terminalRetention,
stopSampler,
signalRoot: retainRootSignaller(
terminal.pid,
@@ -575,7 +581,7 @@ const completeCapture = async (options: {
before: options.before,
after,
truncationDetails: {
raw_terminal: runtime.rawTerminalRetention(),
...runtime.terminalRetention(),
rendered_terminal: runtime.renderer.retention(),
filesystem_before: options.before.coverage,
filesystem_after: after.coverage,
@@ -1,6 +1,7 @@
import { expect, it } from "vitest";
import {
createBoundedProcessTextNormalizer,
createProcessTextNormalizer,
normalizeProcessSamples,
normalizeProcessText,
@@ -95,6 +96,178 @@ it("replaces caller-declared patterns literally, without expanding replacement p
).toBe("listening on $&$&; $$; $1=alpha; tail $1");
});
it("bounds ordered normalization before replacement expansion and preserves later smaller chunks", () => {
const normalize = createBoundedProcessTextNormalizer(
scenario({
paths: false,
ports: false,
pids: false,
patterns: [{ pattern: "a", replacement: "x".repeat(500) }],
}),
"/temporary",
rootPid,
);
expect(normalize("a".repeat(100), 1_000)).toEqual({ state: "over_budget" });
expect(normalize("z", 1_000)).toEqual({
state: "normalized",
value: "z",
utf8Bytes: 1,
});
});
it("keeps ordered intermediate expansion explicit even when a later rule could shrink it", () => {
const normalize = createBoundedProcessTextNormalizer(
scenario({
paths: false,
ports: false,
pids: false,
patterns: [
{ pattern: "a", replacement: "x".repeat(20) },
{ pattern: "x", replacement: "y" },
],
}),
"/temporary",
rootPid,
);
expect(normalize("a", 10)).toEqual({ state: "over_budget" });
});
it("counts UTF-8 bytes across replacement boundaries like the final JavaScript string", () => {
const splitPair = scenario({
paths: false,
ports: false,
pids: false,
patterns: [{ pattern: "\ude00", replacement: "\ude01" }],
});
const splitPairResult = createBoundedProcessTextNormalizer(
splitPair,
"/temporary",
rootPid,
)("😀", 4);
expect(splitPairResult).toEqual({
state: "normalized",
value: normalizeProcessText("😀", splitPair, "/temporary", rootPid),
utf8Bytes: 4,
});
expect(
splitPairResult.state === "normalized" && splitPairResult.utf8Bytes,
).toBe(
Buffer.byteLength(
normalizeProcessText("😀", splitPair, "/temporary", rootPid),
),
);
const joinedPair = scenario({
paths: false,
ports: false,
pids: false,
patterns: [{ pattern: "\ude00\ud83d", replacement: "" }],
});
const joinedInput = "😀😀";
const joinedExpected = normalizeProcessText(
joinedInput,
joinedPair,
"/temporary",
rootPid,
);
expect(joinedExpected).toBe("😀");
expect(
createBoundedProcessTextNormalizer(
joinedPair,
"/temporary",
rootPid,
)(joinedInput, Buffer.byteLength(joinedExpected)),
).toEqual({
state: "normalized",
value: joinedExpected,
utf8Bytes: Buffer.byteLength(joinedExpected),
});
const emptyPattern = scenario({
paths: false,
ports: false,
pids: false,
patterns: [{ pattern: "", replacement: "\ude00" }],
});
const input = "\ud83d";
const expected = normalizeProcessText(
input,
emptyPattern,
"/temporary",
rootPid,
);
expect(
createBoundedProcessTextNormalizer(
emptyPattern,
"/temporary",
rootPid,
)(input, Buffer.byteLength(expected)),
).toEqual({
state: "normalized",
value: expected,
utf8Bytes: Buffer.byteLength(expected),
});
expect(
createBoundedProcessTextNormalizer(
emptyPattern,
"/temporary",
rootPid,
)(input, Buffer.byteLength(expected) - 1),
).toEqual({ state: "over_budget" });
const loneSurrogate = scenario({
paths: false,
ports: false,
pids: false,
patterns: [{ pattern: "x", replacement: "y" }],
});
const loneInput = "\ud83dx";
const loneExpected = normalizeProcessText(
loneInput,
loneSurrogate,
"/temporary",
rootPid,
);
expect(
createBoundedProcessTextNormalizer(
loneSurrogate,
"/temporary",
rootPid,
)(loneInput, Buffer.byteLength(loneExpected)),
).toEqual({
state: "normalized",
value: loneExpected,
utf8Bytes: Buffer.byteLength(loneExpected),
});
});
it("lets a later pattern shrink text when an earlier pattern does not match", () => {
const declared = scenario({
paths: false,
ports: false,
pids: false,
patterns: [
{ pattern: "z", replacement: "x" },
{ pattern: "aa", replacement: "b" },
],
});
const expected = normalizeProcessText("aa", declared, "/temporary", rootPid);
expect(expected).toBe("b");
expect(
createBoundedProcessTextNormalizer(
declared,
"/temporary",
rootPid,
)("aa", Buffer.byteLength(expected)),
).toEqual({
state: "normalized",
value: expected,
utf8Bytes: Buffer.byteLength(expected),
});
});
it("reuses matchers while observing the terminal PID after launch", () => {
let pid = -1;
const normalize = createProcessTextNormalizer(
+303 -21
View File
@@ -3,6 +3,165 @@ import { isIP } from "node:net";
import type { ProcessScenario } from "../../domain/process/processScenario.js";
import type { ProcessSample } from "../../domain/process/processCapture.js";
export type BoundedProcessTextNormalization =
| {
readonly state: "normalized";
readonly value: string;
readonly utf8Bytes: number;
}
| { readonly state: "over_budget" };
type RegexReplacement = (match: string, ...groups: string[]) => string;
type BoundedStageResult =
| {
readonly state: "bounded";
readonly value: string;
readonly utf8Bytes: number;
}
| { readonly state: "unchanged"; readonly value: string }
| { readonly state: "over_budget" };
const createUtf8ByteCounter = (maxBytes: number) => {
let bytes = 0;
let pendingHighSurrogate = false;
const addBytes = (added: number): void => {
if (bytes > maxBytes || added > maxBytes - bytes) {
bytes = Number.POSITIVE_INFINITY;
return;
}
bytes += added;
};
return {
add: (value: string): void => {
if (bytes > maxBytes || value.length === 0) return;
let index = 0;
if (pendingHighSurrogate) {
if (value.length > 0 && isLowSurrogate(value.charCodeAt(0))) {
addBytes(4);
index = 1;
} else {
addBytes(3);
}
pendingHighSurrogate = false;
}
for (; index < value.length; index += 1) {
const code = value.charCodeAt(index);
if (code <= 0x7f) addBytes(1);
else if (code <= 0x7ff) addBytes(2);
else if (isHighSurrogate(code)) {
if (index + 1 === value.length) {
pendingHighSurrogate = true;
} else if (isLowSurrogate(value.charCodeAt(index + 1))) {
addBytes(4);
index += 1;
} else addBytes(3);
} else addBytes(3);
if (bytes > maxBytes) return;
}
},
finish: (): number => {
if (pendingHighSurrogate) addBytes(3);
return bytes;
},
};
};
const isHighSurrogate = (code: number): boolean =>
code >= 0xd800 && code <= 0xdbff;
const isLowSurrogate = (code: number): boolean =>
code >= 0xdc00 && code <= 0xdfff;
const boundedRegexReplace = (
value: string,
matcher: RegExp,
replacement: RegexReplacement,
maxBytes: number,
): BoundedStageResult => {
const byteCounter = createUtf8ByteCounter(maxBytes);
let cursor = 0;
let changed = false;
matcher.lastIndex = 0;
let match: RegExpExecArray | null;
while ((match = matcher.exec(value)) !== null) {
const index = match.index;
const matched = match[0] ?? "";
const nextCursor = index + matched.length;
const replacementValue = replacement(matched, ...match.slice(1));
changed ||= replacementValue !== matched;
byteCounter.add(value.slice(cursor, index));
byteCounter.add(replacementValue);
cursor = nextCursor;
}
byteCounter.add(value.slice(cursor));
const bytes = byteCounter.finish();
if (bytes > maxBytes) {
return changed ? { state: "over_budget" } : { state: "unchanged", value };
}
if (!changed) return { state: "unchanged", value };
matcher.lastIndex = 0;
const normalized = value.replace(matcher, replacement);
return {
state: "bounded",
value: normalized,
utf8Bytes: bytes,
};
};
const boundedLiteralReplace = (
value: string,
pattern: string,
replacement: string,
maxBytes: number,
): BoundedStageResult => {
const byteCounter = createUtf8ByteCounter(maxBytes);
let cursor = 0;
if (pattern === "") {
if (replacement === "") return { state: "unchanged", value };
for (let index = 0; index < value.length; index += 1) {
byteCounter.add(replacement);
byteCounter.add(value.slice(index, index + 1));
}
byteCounter.add(replacement);
const bytes = byteCounter.finish();
if (bytes > maxBytes) return { state: "over_budget" };
return {
state: "bounded",
value: value.replaceAll(pattern, () => replacement),
utf8Bytes: bytes,
};
}
if (pattern === replacement) return { state: "unchanged", value };
let index = value.indexOf(pattern, cursor);
if (index === -1) return { state: "unchanged", value };
while (index !== -1) {
byteCounter.add(value.slice(cursor, index));
byteCounter.add(replacement);
cursor = index + pattern.length;
index = value.indexOf(pattern, cursor);
}
byteCounter.add(value.slice(cursor));
const bytes = byteCounter.finish();
if (bytes > maxBytes) return { state: "over_budget" };
return {
state: "bounded",
value: value.replaceAll(pattern, () => replacement),
utf8Bytes: bytes,
};
};
const boundedIdentity = (
value: string,
maxBytes: number,
): BoundedStageResult => {
const utf8Bytes = Buffer.byteLength(value);
return utf8Bytes <= maxBytes
? { state: "bounded", value, utf8Bytes }
: { state: "unchanged", value };
};
/** Bucket one elapsed process-capture timestamp under scenario normalization. */
export const normalizeProcessElapsedTime = (
elapsedMs: number,
@@ -41,6 +200,84 @@ export const createProcessTextNormalizer = (
};
};
/** Normalize terminal text without materializing an intermediate over its byte budget. */
export const createBoundedProcessTextNormalizer = (
scenario: ProcessScenario,
temporaryRoot: string,
pid: number | (() => number),
): ((value: string, maxBytes: number) => BoundedProcessTextNormalization) => {
const pathRule = scenario.normalization.paths
? createPathRule(scenario, temporaryRoot)
: undefined;
let previousPid: number | undefined;
let pidMatcher: RegExp | undefined;
return (value, maxBytes) => {
let normalized = value;
const apply = (result: BoundedStageResult): boolean => {
if (result.state === "over_budget") return false;
if (result.state === "bounded") normalized = result.value;
return true;
};
if (pathRule !== undefined) {
if (
!apply(
boundedRegexReplace(
normalized,
pathRule.matcher,
pathRule.replace,
maxBytes,
),
)
)
return { state: "over_budget" };
}
if (scenario.normalization.ports) {
if (!apply(normalizePortsBounded(normalized, maxBytes)))
return { state: "over_budget" };
}
if (scenario.normalization.pids) {
const currentPid = typeof pid === "number" ? pid : pid();
if (currentPid !== previousPid) {
previousPid = currentPid;
pidMatcher = createPidMatcher([currentPid]);
}
if (pidMatcher !== undefined) {
if (
!apply(
boundedRegexReplace(
normalized,
pidMatcher,
() => "<pid>",
maxBytes,
),
)
)
return { state: "over_budget" };
}
}
for (const pattern of scenario.normalization.patterns) {
if (
!apply(
boundedLiteralReplace(
normalized,
pattern.pattern,
pattern.replacement,
maxBytes,
),
)
)
return { state: "over_budget" };
}
return boundedIdentity(normalized, maxBytes).state === "bounded"
? {
state: "normalized",
value: normalized,
utf8Bytes: Buffer.byteLength(normalized),
}
: { state: "over_budget" };
};
};
/** Normalize one payload when its caller does not retain a capture-local matcher. */
export const normalizeProcessText = (
value: string,
@@ -50,10 +287,12 @@ export const normalizeProcessText = (
): string => createProcessTextNormalizer(scenario, temporaryRoot, pid)(value);
/** Match original paths once, preferring the most specific selected identity. */
const createPathNormalizer = (
const createPathRule = (
scenario: ProcessScenario,
temporaryRoot: string,
): ((value: string) => string) => {
):
| { readonly matcher: RegExp; readonly replace: RegexReplacement }
| undefined => {
const aliases = new Map<string, string>();
const selected: readonly (readonly [string, string])[] = [
[temporaryRoot, "<temporary-root>"],
@@ -71,12 +310,24 @@ const createPathNormalizer = (
const paths = [...aliases.keys()].sort(
(left, right) => right.length - left.length,
);
if (paths.length === 0) return (value) => value;
if (paths.length === 0) return undefined;
const pattern = paths
.map((path) => path.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&"))
.join("|");
const matcher = new RegExp(pattern, "gu");
return (value) => value.replace(matcher, (path) => aliases.get(path) ?? path);
return {
matcher,
replace: (path) => aliases.get(path) ?? path,
};
};
const createPathNormalizer = (
scenario: ProcessScenario,
temporaryRoot: string,
): ((value: string) => string) => {
const rule = createPathRule(scenario, temporaryRoot);
if (rule === undefined) return (value) => value;
return (value) => value.replace(rule.matcher, rule.replace);
};
// A port may end a sentence, but a period followed by a letter, digit, or
@@ -85,21 +336,45 @@ const createPathNormalizer = (
// PTY chunks are normalized independently, so a period that ends the chunk
// stays unnormalized: its continuation may arrive in the next chunk.
const normalizePortTokens = (value: string): string => {
const endpointNormalized = value.replaceAll(
/(\b[a-z][a-z0-9+.-]*:\/\/[^\s/?#"'<>]+|\blocalhost|\b(?:\d{1,3}\.){3}\d{1,3}|\[[\da-f:.%]+\]):(\d{1,5})(?![\p{L}\p{N}_@:]|\.(?:[\p{L}\p{N}_]|$))/giu,
(match: string, endpoint: string, port: string) => {
if (Number(port) > 65_535) return match;
const isEndpoint = endpoint.includes("://")
? URL.canParse(`${endpoint}:${port}`)
: endpoint.toLowerCase() === "localhost" ||
isIP(endpoint.replace(/^\[|\]$/gu, "")) !== 0;
return isEndpoint ? `${endpoint}:<port>` : match;
},
return value
.replaceAll(PORT_ENDPOINT_MATCHER, portEndpointReplacement)
.replaceAll(PORT_FIELD_MATCHER, portFieldReplacement);
};
const PORT_ENDPOINT_MATCHER =
/(\b[a-z][a-z0-9+.-]*:\/\/[^\s/?#"'<>]+|\blocalhost|\b(?:\d{1,3}\.){3}\d{1,3}|\[[\da-f:.%]+\]):(\d{1,5})(?![\p{L}\p{N}_@:]|\.(?:[\p{L}\p{N}_]|$))/giu;
const PORT_FIELD_MATCHER =
/(\b(?:port|tcp_port|udp_port|listen)\b["']?\s*[:=]\s*)(\d{1,5})(?![\p{L}\p{N}_]|\.(?:[\p{L}\p{N}_]|$))/giu;
const portEndpointReplacement: RegexReplacement = (match, endpoint, port) => {
if (Number(port) > 65_535) return match;
const isEndpoint = endpoint.includes("://")
? URL.canParse(`${endpoint}:${port}`)
: endpoint.toLowerCase() === "localhost" ||
isIP(endpoint.replace(/^\[|\]$/gu, "")) !== 0;
return isEndpoint ? `${endpoint}:<port>` : match;
};
const portFieldReplacement: RegexReplacement = (match, prefix, port) =>
Number(port) <= 65_535 ? `${prefix}<port>` : match;
const normalizePortsBounded = (
value: string,
maxBytes: number,
): BoundedStageResult => {
const endpoint = boundedRegexReplace(
value,
PORT_ENDPOINT_MATCHER,
portEndpointReplacement,
maxBytes,
);
return endpointNormalized.replaceAll(
/(\b(?:port|tcp_port|udp_port|listen)\b["']?\s*[:=]\s*)(\d{1,5})(?![\p{L}\p{N}_]|\.(?:[\p{L}\p{N}_]|$))/giu,
(match: string, prefix: string, port: string) =>
Number(port) <= 65_535 ? `${prefix}<port>` : match,
if (endpoint.state === "over_budget") return endpoint;
const afterEndpoint = endpoint.state === "bounded" ? endpoint.value : value;
return boundedRegexReplace(
afterEndpoint,
PORT_FIELD_MATCHER,
portFieldReplacement,
maxBytes,
);
};
@@ -161,11 +436,18 @@ export const normalizeProcessSamples = (
const createPidNormalizer = (
identifiers: readonly number[],
): ((value: string) => string) => {
const matcher = createPidMatcher(identifiers);
if (matcher === undefined) return (value) => value;
return (value) => value.replace(matcher, "<pid>");
};
const createPidMatcher = (
identifiers: readonly number[],
): RegExp | undefined => {
const tokens = [...new Set(identifiers)]
.filter((identifier) => Number.isSafeInteger(identifier) && identifier > 0)
.sort((left, right) => right - left)
.map(String);
if (tokens.length === 0) return (value) => value;
const matcher = new RegExp(`(?<!\\d)(?:${tokens.join("|")})(?!\\d)`, "gu");
return (value) => value.replace(matcher, "<pid>");
if (tokens.length === 0) return undefined;
return new RegExp(`(?<!\\d)(?:${tokens.join("|")})(?!\\d)`, "gu");
};
+60 -6
View File
@@ -6,6 +6,9 @@ import { expect, it, vi } from "vitest";
import { TerminalRenderer } from "./TerminalRenderer.js";
import { ProcessCaptureResourceScope } from "./ProcessCaptureLifecycle.js";
import { createBoundedProcessTextNormalizer } from "./ProcessNormalization.js";
import { parseProcessScenario } from "../../domain/process/processScenario.js";
import type { BoundedProcessTextNormalization } from "./ProcessNormalization.js";
const require = createRequire(import.meta.url);
const { Terminal } =
@@ -13,6 +16,18 @@ const { Terminal } =
const { SerializeAddon } =
require("@xterm/addon-serialize") as typeof import("@xterm/addon-serialize");
const boundedNormalize =
(
normalize: (value: string) => string,
): ((value: string, maxBytes: number) => BoundedProcessTextNormalization) =>
(value, maxBytes) => {
const normalized = normalize(value);
const utf8Bytes = Buffer.byteLength(normalized);
return utf8Bytes <= maxBytes
? { state: "normalized", value: normalized, utf8Bytes }
: { state: "over_budget" };
};
const renderFrames = async (
data: string,
normalize: (value: string) => string = (value) => value,
@@ -22,7 +37,7 @@ const renderFrames = async (
rows: 4,
scrollback: 10,
maxBytes: 100_000,
normalize,
normalize: boundedNormalize(normalize),
});
renderer.write(data, 0);
const frames = await renderer.frames();
@@ -56,7 +71,7 @@ it("applies rendered retention cumulatively across sparse frames", async () => {
rows: 1,
scrollback: 0,
maxBytes: 5,
normalize: (value) => value,
normalize: boundedNormalize((value) => value),
});
try {
renderer.write("a", 0);
@@ -66,10 +81,11 @@ it("applies rendered retention cumulatively across sparse frames", async () => {
expect(frames.map(({ lines }) => lines)).toEqual([["a"]]);
expect(renderer.retention()).toEqual({
budget_bytes: 5,
observed_bytes: 6,
observed_bytes: 2,
retained_bytes: 2,
observed_frames: 2,
retained_frames: 1,
unknown_size_frames: 1,
});
expect(renderer.truncated()).toBe(true);
} finally {
@@ -84,6 +100,44 @@ it("trims only after normalization sees the full-width row", async () => {
expect(frames[0]?.lines[0]).toBe("pid <pid>");
});
it("omits rendered frames with unknown expanded size and keeps later bounded states", async () => {
const renderer = new TerminalRenderer({
columns: 10,
rows: 1,
scrollback: 0,
maxBytes: 1_000,
normalize: createBoundedProcessTextNormalizer(
parseProcessScenario({
executable: "unused",
normalization: {
paths: false,
pids: false,
ports: false,
patterns: [{ pattern: "a", replacement: "x".repeat(2_000) }],
},
}),
"/tmp",
-1,
),
});
try {
renderer.write("a", 0);
renderer.write("\u001b[2J\rz", 1);
const frames = await renderer.frames();
expect(frames).toHaveLength(1);
expect(frames[0]?.lines[0]).toBe("z");
expect(renderer.retention()).toMatchObject({
observed_frames: 2,
retained_frames: 1,
unknown_size_frames: 1,
});
expect(renderer.truncated()).toBe(true);
} finally {
await renderer.dispose();
}
});
it.each(["write", "resize"] as const)(
"releases terminal resources after a queued %s observation fails",
async (operation) => {
@@ -95,9 +149,9 @@ it.each(["write", "resize"] as const)(
rows: 4,
scrollback: 10,
maxBytes: 100_000,
normalize: () => {
normalize: boundedNormalize(() => {
throw failure;
},
}),
});
const scope = new ProcessCaptureResourceScope();
const temporaryRoot = await mkdtemp(
@@ -147,7 +201,7 @@ it("retains disposal failures for a later cleanup retry", async () => {
rows: 4,
scrollback: 10,
maxBytes: 100_000,
normalize: (value) => value,
normalize: boundedNormalize((value) => value),
});
const scope = new ProcessCaptureResourceScope();
const temporaryRoot = await mkdtemp(join(tmpdir(), "rea-terminal-retry-"));
+44 -18
View File
@@ -1,5 +1,6 @@
import { createRequire } from "node:module";
import type { TerminalRetention } from "../../domain/process/processCaptureCoverage.js";
import type { BoundedProcessTextNormalization } from "./ProcessNormalization.js";
import type {
RecordProcessCaptureEvent,
@@ -21,7 +22,10 @@ interface TerminalRendererOptions {
readonly rows: number;
readonly scrollback: number;
readonly maxBytes: number;
readonly normalize: (value: string) => string;
readonly normalize: (
value: string,
maxBytes: number,
) => BoundedProcessTextNormalization;
readonly recordEvent?: RecordProcessCaptureEvent;
}
@@ -42,6 +46,7 @@ export class TerminalRenderer {
#observedBytes = 0;
#observedFrames = 0;
#truncated = false;
#unknownSizeFrames = 0;
constructor(private readonly options: TerminalRendererOptions) {
this.#terminal = new HeadlessPackage.Terminal({
@@ -97,6 +102,9 @@ export class TerminalRenderer {
retained_bytes: this.#capturedBytes,
observed_frames: this.#observedFrames,
retained_frames: this.#frames.length,
...(this.#unknownSizeFrames === 0
? {}
: { unknown_size_frames: this.#unknownSizeFrames }),
};
}
@@ -118,32 +126,50 @@ export class TerminalRenderer {
#capture(atMs: number): void {
const buffer = this.#terminal.buffer.active;
const lines: string[] = [];
this.#observedFrames += 1;
let remainingBytes = this.options.maxBytes - this.#capturedBytes;
let bytes = 0;
for (let row = 0; row < this.#terminal.rows; row += 1) {
const line = buffer.getLine(buffer.viewportY + row);
const paddedLine = (
line?.translateToString(false, 0, this.#terminal.cols) ?? ""
).padEnd(this.#terminal.cols, " ");
const trailingSpaceBytes = Buffer.byteLength(
paddedLine.match(TRAILING_SPACES)?.[0] ?? "",
);
const normalized = this.options.normalize(
paddedLine,
Math.min(Number.MAX_SAFE_INTEGER, remainingBytes + trailingSpaceBytes),
);
if (normalized.state === "over_budget") {
this.#unknownSizeFrames += 1;
this.#truncated = true;
return;
}
const normalizedLine = normalized.value.replace(TRAILING_SPACES, "");
const lineBytes = Buffer.byteLength(normalizedLine);
if (lineBytes > remainingBytes) {
this.#unknownSizeFrames += 1;
this.#truncated = true;
return;
}
remainingBytes -= lineBytes;
bytes += lineBytes;
// Trailing blank cells carry no information beyond `columns`; dropping
// them keeps sparse screens from costing a full row per blank line.
lines.push(
this.options
.normalize(
(
line?.translateToString(false, 0, this.#terminal.cols) ?? ""
).padEnd(this.#terminal.cols, " "),
)
.replace(TRAILING_SPACES, ""),
);
lines.push(normalizedLine);
}
const serializedState = this.options.normalize(
const normalizedState = this.options.normalize(
this.#serializeAddon.serialize(),
remainingBytes,
);
const bytes =
Buffer.byteLength(serializedState) +
lines.reduce((total, line) => total + Buffer.byteLength(line), 0);
this.#observedBytes += bytes;
this.#observedFrames += 1;
if (this.#capturedBytes + bytes > this.options.maxBytes) {
if (normalizedState.state === "over_budget") {
this.#unknownSizeFrames += 1;
this.#truncated = true;
return;
}
bytes += normalizedState.utf8Bytes;
this.#observedBytes += bytes;
this.#capturedBytes += bytes;
const sequence = this.#frames.length;
this.#frames.push({
@@ -155,7 +181,7 @@ export class TerminalRenderer {
cursor_y: buffer.cursorY,
active_buffer: buffer.type,
lines,
serialized_state: serializedState,
serialized_state: normalizedState.value,
});
this.options.recordEvent?.("rendered_frames", sequence);
}
+123 -1
View File
@@ -1,6 +1,25 @@
import { describe, expect, it } from "vitest";
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
import { z } from "zod";
import { forgetPendingMcpRequest } from "./EvidenceMcpServer.js";
import { InvestigationRecords } from "../application/investigation/InvestigationRecords.js";
import {
EvidenceMcpServer,
forgetPendingMcpRequest,
} from "./EvidenceMcpServer.js";
import { ToolResultDelivery } from "./toolResult.js";
const firstText = (result: {
readonly content: readonly {
readonly type: string;
readonly text?: string;
}[];
}): string | undefined => {
for (const item of result.content) {
if (item.type === "text") return item.text;
}
return undefined;
};
describe("MCP request bookkeeping", () => {
it("releases list, tool, and completion state when cancellation arrives", () => {
@@ -22,3 +41,106 @@ describe("MCP request bookkeeping", () => {
expect(pendingCompletions.has(7)).toBe(false);
});
});
describe("SDK validation error delivery", () => {
it("bounds oversized errors, retains their source, and preserves fitting and valid calls", async () => {
const delivery = new ToolResultDelivery(4096);
const records = new InvestigationRecords();
const server = new EvidenceMcpServer(
{ name: "sdk-validation-budget-test", version: "1" },
{ capabilities: {} },
(evidence) => records.recordEvidence(evidence),
delivery,
);
server.registerTool(
"procedure_pseudo_code",
{ inputSchema: z.strictObject({ required: z.string() }) },
async () => ({ content: [{ type: "text", text: "valid call" }] }),
);
const client = new Client({
name: "sdk-validation-budget-client",
version: "1",
});
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
const oversizedKey = `unexpected_${"x".repeat(6000)}`;
try {
await server.connect(serverTransport);
await client.connect(clientTransport);
const oversized = await client.callTool({
name: "procedure_pseudo_code",
arguments: { required: "present", [oversizedKey]: true },
});
expect(oversized.isError).toBe(true);
expect(oversized.structuredContent).toBeUndefined();
expect(oversized._meta).toEqual({
provenance: { untrusted_content: true, source: "analysis_target" },
});
expect(Buffer.byteLength(JSON.stringify(oversized))).toBeLessThan(4096);
const oversizedError = z
.object({
error: z.object({
code: z.string(),
details: z.object({
reported_limits: z.record(z.string(), z.unknown()),
}),
}),
})
.parse(JSON.parse(firstText(oversized) ?? "null"));
expect(oversizedError.error.code).toBe("resource_constraint");
const reportedLimits = oversizedError.error.details.reported_limits;
expect(reportedLimits).toMatchObject({
evidence_reference: {
kind: "retained-evidence",
evidence_id: expect.any(String),
},
diagnostic_truncated: true,
});
expect(reportedLimits).not.toHaveProperty("retention");
const bundle = records.exportEvidenceBundle();
expect(bundle.records).toHaveLength(1);
expect(bundle.records[0]?.operation).toBe("mcp_tool_error");
expect(bundle.records[0]?.raw_result).toMatchObject({
isError: true,
content: [
{ type: "text", text: expect.stringContaining(oversizedKey) },
],
});
const evidenceReference = z
.object({ evidence_id: z.string() })
.parse(reportedLimits.evidence_reference);
expect(bundle.records[0]?.evidence_id).toBe(
evidenceReference.evidence_id,
);
const fitting = await client.callTool({
name: "procedure_pseudo_code",
arguments: { unexpected: true },
});
expect(fitting.isError).toBe(true);
expect(firstText(fitting)).toContain("Input validation error");
const quotedKey = '"\\'.repeat(350);
const quoted = await client.callTool({
name: "procedure_pseudo_code",
arguments: { required: "present", [quotedKey]: true },
});
expect(quoted.isError).toBe(true);
expect(firstText(quoted)).toContain("Input validation error");
expect(Buffer.byteLength(JSON.stringify(quoted))).toBeLessThan(4096);
expect(records.exportEvidenceBundle().records).toHaveLength(1);
const valid = await client.callTool({
name: "procedure_pseudo_code",
arguments: { required: "present" },
});
expect(valid.content).toContainEqual({
type: "text",
text: "valid call",
});
} finally {
await Promise.all([client.close(), server.close()]);
}
});
});
+54 -20
View File
@@ -37,6 +37,7 @@ import { toolReturnsTargetContent } from "../contracts/toolEffects.js";
import {
carriesTargetContentMarker,
encodeToolResult,
measureMcpResultWithinBudget,
TARGET_CONTENT_RESULT_META,
} from "./toolResultEncoding.js";
@@ -346,38 +347,65 @@ class ErrorEvidenceTransport implements Transport {
if (
!isJSONRPCResultResponse(message) ||
!isCallToolResult(message.result) ||
message.result.isError !== true ||
message.result.structuredContent === undefined
message.result.isError !== true
)
return message;
const structured = jsonObjectSchema.parse(message.result.structuredContent);
const budget = this.delivery.resultBudgetBytes;
// Count the provenance marker exactly as the projected error did, and keep
// it on the replacement: the retained original may hold target content.
const targetContent = carriesTargetContentMarker(message.result);
// The delivered error omits structuredContent, so its text and error flag count.
const encoded = encodeToolResult(
structured,
budget,
"error-text",
targetContent,
);
const targetContent =
carriesTargetContentMarker(message.result) ||
(tool !== undefined && toolReturnsTargetContent(tool));
// SDK validation failures have no structured carrier. Size their existing
// text-only result without serializing the complete JSON-RPC message.
const isStructured = message.result.structuredContent !== undefined;
const structured = isStructured
? jsonObjectSchema.parse(message.result.structuredContent)
: undefined;
const sdkResultWithMarker =
!isStructured &&
targetContent &&
!carriesTargetContentMarker(message.result)
? {
...message.result,
_meta: {
...message.result._meta,
...TARGET_CONTENT_RESULT_META,
},
}
: message.result;
const encoded =
structured !== undefined
? encodeToolResult(structured, budget, "error-text", targetContent)
: measureMcpResultWithinBudget(sdkResultWithMarker, budget);
if (encoded.ok) return message;
const operation = tool ?? "mcp_tool_error";
const originalError = analysisErrorProjectionSchema.safeParse(
structured["error"],
);
// Retain the already projected diagnostic. Transport authentication and raw
// request arguments are deliberately excluded: the application owns their
// explicit sensitivity policy, and its error projection is authoritative.
const originalResult = jsonObjectSchema.parse(message.result);
const sdkDiagnostic = !isStructured
? message.result.content.find((item) => item.type === "text")?.text
: undefined;
const retainedResult = structured ?? originalResult;
const originalError = isStructured
? analysisErrorProjectionSchema.safeParse(structured?.["error"])
: undefined;
// Retain the complete original response, including SDK validation text.
// The SDK error contains its rendered issue, not a separate argument bag;
// transport authentication and caller arguments are never added here.
const evidence = createEvidence(
undefined,
{ id: "rea-mcp", name: "REA MCP delivery", version: "1" },
{
operation: "mcp_tool_error",
parameters: { tool_name: operation },
result: structured,
rawResult: jsonObjectSchema.parse(message.result),
result:
sdkDiagnostic === undefined
? retainedResult
: {
boundary: "sdk-input-validation",
diagnostic_preview: sdkDiagnostic.slice(0, 256),
diagnostic_truncated: sdkDiagnostic.length > 256,
},
rawResult: originalResult,
},
);
const recorded = this.recordEvidence?.(evidence);
@@ -388,12 +416,18 @@ class ErrorEvidenceTransport implements Transport {
"The tool failed, but its complete error response exceeds the MCP response budget.",
{
boundary: "mcp-response",
...(originalError.success
...(originalError?.success
? { original_error_code: originalError.data.code }
: {}),
result_budget_bytes: budget,
response_bytes_at_least: encoded.bytesAtLeast,
constraint: encoded.constraint,
...(sdkDiagnostic === undefined
? {}
: {
diagnostic_preview: sdkDiagnostic.slice(0, 256),
diagnostic_truncated: sdkDiagnostic.length > 256,
}),
...(retained
? {
evidence_reference: {
+4
View File
@@ -577,6 +577,10 @@ export const createServer = (
: { availabilityPolicy: availability.policy }),
androidAnalysisAvailability: (signal) =>
android.inspectAvailability(signal),
adbAnalysisAvailability: (signal) =>
adbDevice.inspectAvailability(signal),
apktoolAnalysisAvailability: (signal) =>
apktool.inspectAvailability(signal),
startedAt,
});
}
+24 -3
View File
@@ -30,6 +30,12 @@ export interface SessionStatusToolOptions {
readonly androidAnalysisAvailability: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly adbAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly apktoolAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
}
/** Register the read-only provider and target status operation. */
@@ -43,6 +49,8 @@ export const registerSessionStatusTool = (
startedAt,
availabilityPolicy,
androidAnalysisAvailability,
adbAnalysisAvailability,
apktoolAnalysisAvailability,
} = options;
server.registerTool(
contract.name,
@@ -53,10 +61,17 @@ export const registerSessionStatusTool = (
server.server,
);
let androidAvailability: ProviderAvailability;
let adbAvailability: ProviderAvailability | undefined;
let apktoolAvailability: ProviderAvailability | undefined;
try {
androidAvailability = await androidAnalysisAvailability(
context.mcpReq.signal,
);
const [android, adb, apktool] = await Promise.all([
androidAnalysisAvailability(context.mcpReq.signal),
adbAnalysisAvailability?.(context.mcpReq.signal) ?? undefined,
apktoolAnalysisAvailability?.(context.mcpReq.signal) ?? undefined,
] as const);
androidAvailability = android;
adbAvailability = adb;
apktoolAvailability = apktool;
} catch (cause) {
if (!context.mcpReq.signal.aborted) throw cause;
return server.delivery.toCallToolResult(
@@ -71,6 +86,12 @@ export const registerSessionStatusTool = (
{
...availabilityPolicy(),
androidAnalysisAvailability: androidAvailability,
...(adbAvailability === undefined
? {}
: { adbAnalysisAvailability: adbAvailability }),
...(apktoolAvailability === undefined
? {}
: { apktoolAnalysisAvailability: apktoolAvailability }),
},
clientFeatures,
);
+26
View File
@@ -155,6 +155,12 @@ export interface LifecycleToolRegistration {
readonly androidAnalysisAvailability: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly adbAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly apktoolAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly closeProcessResources: () => Promise<Result<null, AnalysisError>>;
}
@@ -169,6 +175,8 @@ const registerLifecycleTools = (
startedAt,
availabilityPolicy,
androidAnalysisAvailability,
adbAnalysisAvailability,
apktoolAnalysisAvailability,
} = registration;
registerOpenLifecycleTool(registration, artifactResources);
registerCloseLifecycleTool(registration);
@@ -179,6 +187,12 @@ const registerLifecycleTools = (
startedAt,
availabilityPolicy,
androidAnalysisAvailability,
...(adbAnalysisAvailability === undefined
? {}
: { adbAnalysisAvailability }),
...(apktoolAnalysisAvailability === undefined
? {}
: { apktoolAnalysisAvailability }),
});
};
@@ -238,6 +252,12 @@ export interface SessionToolOptions {
readonly androidAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly adbAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
readonly apktoolAnalysisAvailability?: (
signal: AbortSignal,
) => Promise<ProviderAvailability>;
}
const registerContextTools = (
@@ -317,6 +337,12 @@ export const registerSessionTools = (
),
androidAnalysisAvailability:
options.androidAnalysisAvailability ?? missingAndroidAvailability,
...(options.adbAnalysisAvailability === undefined
? {}
: { adbAnalysisAvailability: options.adbAnalysisAvailability }),
...(options.apktoolAnalysisAvailability === undefined
? {}
: { apktoolAnalysisAvailability: options.apktoolAnalysisAvailability }),
closeProcessResources: async () => {
try {
await processResources.retryCleanup();
+20
View File
@@ -63,6 +63,26 @@ export type ToolResultEncoding =
readonly constraint: "receive-buffer" | "string-length";
};
/** Measure an already-formed MCP result object without allocating its JSON text. */
export const measureMcpResultWithinBudget = (
value: unknown,
budgetBytes: number,
):
| { readonly ok: true; readonly bytes: number }
| {
readonly ok: false;
readonly bytesAtLeast: number;
readonly constraint: "receive-buffer";
} => {
let bytes = 0;
for (const part of bufferedJsonParts(jsonParts(value))) {
bytes += Buffer.byteLength(part);
if (bytes > budgetBytes)
return { ok: false, bytesAtLeast: bytes, constraint: "receive-buffer" };
}
return { ok: true, bytes };
};
/** Account for the delivered JSON representations before allocating the complete text. */
export const encodeToolResult = (
value: JsonValue,
@@ -308,6 +308,47 @@ cliTest(
);
for (const surface of ["CLI", "stdio MCP"] as const) {
cliTest(
`retains explicit nested contextBridge members after an unknown spread through ${surface}`,
async ({ cli }) => {
const analyzeBridge = async (api: string) => {
const root = await createTestTempDirectory(
"rea-electron-spread-bridge-members-",
);
await writeFixtureFiles(root, {
"preload.js": `const { contextBridge } = require("electron");
const extra = getExtra();
contextBridge.exposeInMainWorld("api", ${api});`,
});
return analyzeJavaScriptDirectory(surface, cli, root);
};
const withSpread = await analyzeBridge("{ ...extra, a: { b: 1 }, c: 2 }");
const control = await analyzeBridge("{ a: { b: 1 }, c: 2 }");
const bridgeObservation = (application: typeof withSpread) =>
application.graph.nodes
.filter((node) => node.kind === "context-bridge-api")
.flatMap((node) => node.observations)
.find((observation) => observation.properties.api_key === "api");
expect(bridgeObservation(withSpread)).toMatchObject({
properties: {
api_status: "dynamic",
members: ["a", "a.b", "c"],
unknown_members: 1,
},
});
expect(bridgeObservation(control)).toMatchObject({
properties: {
api_status: "object-literal",
members: ["a", "a.b", "c"],
unknown_members: 0,
},
});
expect(withSpread.summary.exposed_api_members).toBe(3);
expect(control.summary.exposed_api_members).toBe(3);
},
);
cliTest(
`preserves handler ambiguity and short-circuit factory paths through ${surface}`,
async ({ cli }) => {
@@ -1,8 +1,12 @@
import { rm } from "node:fs/promises";
import { join } from "node:path";
import { expect } from "vitest";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { createStrippedAsarAddon } from "../../fixtures/strippedAsarAddon.js";
import { cliTest } from "../../support/cli/cliFixture.js";
import {
findExportNode,
@@ -11,6 +15,75 @@ import {
writeFixtureFiles,
} from "../../support/javascriptApplicationFixture.js";
cliTest(
"resolves ESM specifiers containing whitespace using URL text semantics",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-esm-whitespace-path-");
const rawSpecifier = "./feature\tname.js";
const encodedSpecifier = "./feature%09name.js";
await writeFixtureFiles(root, {
"main.mjs": `import ${JSON.stringify(rawSpecifier)};\nimport ${JSON.stringify(encodedSpecifier)};\n`,
"featurename.js": "export const plain = 1;\n",
"feature\tname.js": "export const value = 1;\n",
});
const output = await cli.run({
arguments: ["analyze-javascript-application", root, "--json"],
});
expect(output.exitCode, output.stderr).toBe(0);
const result = javascriptApplicationAnalysisResultSchema.parse(
parseEvidence(output.json).normalized_result,
);
expect(
findGraphEdge(
result.graph,
findSourceModule(result.graph, "main.mjs"),
findSourceModule(result.graph, "featurename.js"),
"imports",
)?.properties,
).toMatchObject({
specifier: rawSpecifier,
resolved_path: "featurename.js",
});
expect(
findGraphEdge(
result.graph,
findSourceModule(result.graph, "main.mjs"),
findSourceModule(result.graph, "feature\tname.js"),
"imports",
)?.properties,
).toMatchObject({
specifier: encodedSpecifier,
resolved_path: "feature\tname.js",
});
},
);
cliTest(
"retains an inventoried native addon when its ASAR companion is missing",
async ({ cli }) => {
const { archive, addon } = await createStrippedAsarAddon();
await rm(join(`${archive}.unpacked`, addon));
const output = await cli.run({
arguments: ["analyze-javascript-application", archive, "--json"],
});
expect(output.exitCode, output.stderr).toBe(0);
const result = javascriptApplicationAnalysisResultSchema.parse(
parseEvidence(output.json).normalized_result,
);
expect(result.graph.coverage.status).toBe("partial");
expect(
result.graph.nodes.some(
(node) =>
node.kind === "native-addon" &&
node.observations.some(
({ label, properties }) =>
label === addon && properties.availability === "unavailable",
),
),
).toBe(true);
},
);
cliTest(
"resolves emitted module imports without substituting Electron runtime paths",
async ({ cli }) => {
@@ -1009,6 +1009,10 @@ it.each([
".github/workflows/release.yml",
{ workflows: true, code: true, package: false, website: false },
],
[
".github/workflows/pr-readiness.yml",
{ workflows: true, code: true, package: false, website: false },
],
[
".github/workflows/real-ghidra-windows.yml",
{ workflows: true, code: true, package: false, website: false },
@@ -16,6 +16,7 @@ import {
} from "../../../src/application/SetupClientConfiguration.js";
import { systemUninstallHost } from "../../../src/application/Uninstall.js";
import { supportedClients } from "../../../src/application/SupportedClients.js";
import { CURRENT_ENTRYPOINT_PATH } from "../../../src/process/CurrentEntrypointPath.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
describe.each([
@@ -137,8 +138,10 @@ it("restores original BOM-prefixed JSONC bytes after update and removal", async
await mkdir(dirname(client.configPath), { recursive: true });
await writeFile(client.configPath, original);
const firstCommand = [join(home, "bin", "rea"), "mcp"];
const updatedCommand = [join(home, "updated", "bin", "rea"), "mcp"];
if (CURRENT_ENTRYPOINT_PATH === undefined)
throw new Error("test runner entrypoint is unavailable");
const firstCommand = ["rea", "mcp"];
const updatedCommand = [process.execPath, CURRENT_ENTRYPOINT_PATH, "mcp"];
expect(
await configureClientConfiguration(client, {}, firstCommand),
).toMatchObject({ status: "configured" });
@@ -1335,7 +1335,7 @@ describe("Codex configuration comments", () => {
await configureClientConfiguration(
{ name: "codex", configPath, format: "toml" },
{},
["/isolated prefix/bin/rea", "mcp"],
[PRODUCT_IDENTITY.cliBinary, "mcp"],
),
).toMatchObject({ status: "configured" });
const configured = await readFile(configPath, "utf8");
@@ -1505,7 +1505,7 @@ describe("client configuration filesystem removal", () => {
cursor,
JSON.stringify({
mcpServers: {
rea: { command: "/isolated prefix/bin/rea", args: ["mcp"] },
rea: { command: PRODUCT_IDENTITY.cliBinary, args: ["mcp"] },
other: { command: "other" },
},
}),
@@ -502,10 +502,72 @@ it("keeps ASAR analysis usable when unpacked native companion bytes are absent",
"ASAR unpacked companion bytes were unavailable; no content hash or child artifact was produced.",
]),
});
if (missingNative === undefined)
throw new Error("Inventory omitted the unavailable native companion");
await reconstructAsarArchive(archive, [
const result = await reconstructAsarArchive(archive, [
expect.objectContaining({ kind: "javascript-asset" }),
]);
const missingEntry = result.graph.nodes.find(
({ kind, observations }) =>
kind === "asar-entry" &&
observations.some(({ label }) => label === missingNative.logical_path),
);
expect(missingEntry).toMatchObject({
identity: {
strategy: "canonical-path",
stability: "artifact-version",
artifact_sha256: snapshot.manifest.root_sha256,
path: missingNative.logical_path,
},
observations: [
expect.objectContaining({
label: missingNative.logical_path,
properties: expect.objectContaining({
entry_sha256: null,
bytes: missingNative.declared_size,
inventory_artifact_id: null,
inventory_occurrence_id: missingNative.occurrence_id,
hash_status: "unavailable",
artifact_kind: "native-addon",
kind: "native-addon",
content_available: false,
}),
evidence: expect.objectContaining({
limitations: expect.arrayContaining([
"ASAR unpacked companion bytes were unavailable; no content hash or child artifact was produced.",
"The inventoried native-addon entry has unavailable bytes; no content artifact identity can be established.",
]),
}),
}),
],
});
const unavailableAddon = result.graph.nodes.find(
({ kind, observations }) =>
kind === "native-addon" &&
observations.some(({ label }) => label === missingNative.logical_path),
);
expect(unavailableAddon).toMatchObject({
identity: {
strategy: "canonical-path",
stability: "artifact-version",
artifact_sha256: snapshot.manifest.root_sha256,
path: missingNative.logical_path,
},
observations: [
expect.objectContaining({
label: missingNative.logical_path,
properties: expect.objectContaining({
availability: "unavailable",
hash_status: "unavailable",
artifact_kind: "native-addon",
bytes: missingNative.declared_size,
inventory_artifact_id: null,
inventory_occurrence_id: missingNative.occurrence_id,
}),
}),
],
});
});
it("recurses into filesystem-backed ASAR containers without losing container-relative paths", async () => {
@@ -0,0 +1,382 @@
import { execFile } from "node:child_process";
import { readFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { describe, expect, it, vi } from "vitest";
import { parse } from "yaml";
import { z } from "zod";
import {
evaluateReadiness,
runReadinessCheck,
type ReadinessGitHub,
type ReadinessPullRequest,
} from "../../../scripts/ci/pr-readiness.mjs";
const template = await readFile(
new URL("../../../.github/pull_request_template.md", import.meta.url),
"utf8",
);
const readyBody = template
.slice(
template.indexOf("## Author readiness"),
template.indexOf("## Before merge"),
)
.replace("Human owner: <!-- @GitHub-handle -->", "Human owner: @contributor")
.replaceAll("- [ ]", "- [x]");
const makePull = (
changes: Partial<ReadinessPullRequest> = {},
): ReadinessPullRequest => ({
number: 42,
state: "open",
draft: false,
body: readyBody,
head: { sha: "a".repeat(40) },
base: { ref: "main" },
...changes,
});
function fixture(changes: Partial<ReadinessPullRequest> = {}) {
const pull = makePull(changes);
const get = vi.fn(async () => ({ data: pull }));
const createCommitStatus = vi.fn(
async (
_status: Parameters<
ReadinessGitHub["rest"]["repos"]["createCommitStatus"]
>[0],
) => ({}),
);
const paginate = vi.fn(async () => [pull]);
const github = {
rest: {
pulls: { get },
repos: {
createCommitStatus,
listPullRequestsAssociatedWithCommit: vi.fn(),
},
},
paginate,
};
const context = {
repo: { owner: "morluto", repo: "rea" },
payload: { pull_request: { number: 42 } },
serverUrl: "https://github.com",
runId: 123,
};
const core = { info: vi.fn(), setFailed: vi.fn() };
return { github, context, core, get, createCommitStatus, paginate, pull };
}
describe("initial author declaration", () => {
it("accepts the filled current template and uppercase checkmarks", () => {
expect(evaluateReadiness(makePull(), template).state).toBe("success");
expect(
evaluateReadiness(
makePull({ body: readyBody.replaceAll("[x]", "[X]") }),
template,
).state,
).toBe("success");
expect(
evaluateReadiness(
makePull({ body: readyBody.replaceAll("\n", "\r\n") }),
template,
).state,
).toBe("success");
});
it("allows drafts without a body or declarations", () => {
expect(
evaluateReadiness(makePull({ draft: true, body: null }), template).state,
).toBe("pending");
});
it.each([0, 1, 2])("requires declaration %i to be checked", (index) => {
let position = 0;
const body = readyBody.replaceAll("- [x]", () =>
position++ === index ? "- [ ]" : "- [x]",
);
expect(evaluateReadiness(makePull({ body }), template).state).toBe(
"failure",
);
});
it.each([
null,
readyBody.replace("Human owner: @contributor", "Human owner:"),
readyBody.replace("Human owner: @contributor", "Human owner: @one @two"),
readyBody.replace(
"Human owner: @contributor",
"Human owner: @automation[bot]",
),
readyBody + "\nHuman owner: @other\n",
readyBody.replace("## Author readiness", "## Other"),
readyBody + readyBody,
`<!--\n${readyBody}\n-->`,
`~~~markdown\n${readyBody}\n~~~`,
`\`\`\`markdown\n${readyBody}\n\`\`\``,
readyBody
.split("\n")
.map((line) => `> ${line}`)
.join("\n"),
readyBody.replace(
"- [x] I declare this PR ready for initial maintainer review.",
"",
),
readyBody +
"\n- [x] I declare this PR ready for initial maintainer review.\n",
])(
"rejects missing, hidden, ambiguous, or incomplete declarations (%#)",
(body) => {
expect(evaluateReadiness(makePull({ body }), template).state).toBe(
"failure",
);
},
);
it("ignores checked examples in comments and fenced blocks before the actual declaration", () => {
for (const example of [
`<!--\n${readyBody}\n-->\n`,
`~~~markdown\n${readyBody}\n~~~\n`,
`\`\`\`markdown\n${readyBody}\n\`\`\`\n`,
])
expect(
evaluateReadiness(makePull({ body: example + readyBody }), template)
.state,
).toBe("success");
});
it("does not treat an HTML comment inside a code example as hiding the real declaration", () => {
const body =
"~~~html\n<!-- an intentionally unfinished HTML example\n~~~\n" +
readyBody;
expect(evaluateReadiness(makePull({ body }), template).state).toBe(
"success",
);
});
it("fails closed when the trusted template loses a declaration", () => {
expect(() =>
evaluateReadiness(
makePull(),
template.replace(
"- [ ] I declare this PR ready for initial maintainer review.",
"",
),
),
).toThrow("three readiness declarations");
});
});
describe("GitHub status boundary", () => {
it("publishes failure on the actual PR head when declarations are unchecked", async () => {
const input = fixture({ body: readyBody.replaceAll("[x]", "[ ]") });
expect((await runReadinessCheck(input))?.state).toBe("failure");
expect(
input.createCommitStatus.mock.calls.map(([status]) => status),
).toEqual([
expect.objectContaining({
sha: input.pull.head.sha,
context: "PR readiness",
state: "pending",
}),
expect.objectContaining({
sha: input.pull.head.sha,
context: "PR readiness",
state: "failure",
}),
]);
expect(input.core.setFailed).toHaveBeenCalledOnce();
});
it("keeps drafts pending without failing the workflow", async () => {
const input = fixture({ draft: true, body: null });
expect((await runReadinessCheck(input))?.state).toBe("pending");
expect(input.core.setFailed).not.toHaveBeenCalled();
expect(input.paginate).not.toHaveBeenCalled();
});
it("keeps the initial declaration valid after later pushes", async () => {
for (const sha of ["a".repeat(40), "b".repeat(40)]) {
const input = fixture({ head: { sha } });
expect((await runReadinessCheck(input))?.state).toBe("success");
expect(input.createCommitStatus).toHaveBeenLastCalledWith(
expect.objectContaining({ sha, state: "success" }),
);
expect(input.github.rest.pulls).toEqual({ get: input.get });
}
});
it("does not publish stale success if the PR changes during the check", async () => {
const input = fixture();
input.get
.mockResolvedValueOnce({ data: input.pull })
.mockResolvedValueOnce({
data: makePull({ head: { sha: "b".repeat(40) }, body: null }),
});
expect((await runReadinessCheck(input))?.state).toBe("pending");
expect(input.createCommitStatus).toHaveBeenLastCalledWith(
expect.objectContaining({ sha: "b".repeat(40), state: "pending" }),
);
});
it("prevents another open PR from lending its status to the same head", async () => {
const input = fixture();
input.paginate.mockResolvedValue([input.pull, makePull({ number: 43 })]);
expect((await runReadinessCheck(input))?.state).toBe("failure");
expect(input.core.setFailed).toHaveBeenCalledWith(
expect.stringContaining("distinct head commits"),
);
});
it("leaves the status pending and surfaces API failure", async () => {
const input = fixture();
input.paginate.mockRejectedValue(new Error("GitHub unavailable"));
await expect(runReadinessCheck(input)).rejects.toThrow(
"GitHub unavailable",
);
expect(input.createCommitStatus).toHaveBeenCalledOnce();
expect(input.createCommitStatus).toHaveBeenCalledWith(
expect.objectContaining({ state: "pending" }),
);
});
it("keeps readiness unknown when GitHub has not associated the head with this PR", async () => {
const input = fixture();
input.paginate.mockResolvedValue([]);
expect((await runReadinessCheck(input))?.state).toBe("pending");
expect(input.core.setFailed).not.toHaveBeenCalled();
});
it.each([{ state: "closed" as const }, { base: { ref: "other-branch" } }])(
"ignores closed PRs and unprotected target branches (%#)",
async (changes) => {
const input = fixture(changes);
await runReadinessCheck(input);
expect(input.createCommitStatus).not.toHaveBeenCalled();
},
);
it("supports manually rechecking an existing PR", async () => {
const input = fixture();
await runReadinessCheck({
...input,
context: {
...input.context,
payload: { inputs: { pull_request_number: "42" } },
},
});
expect(input.get).toHaveBeenCalledWith({
owner: "morluto",
repo: "rea",
pull_number: 42,
});
});
});
const readWorkflow = async () =>
z
.object({
on: z.object({
pull_request_target: z.object({
branches: z.array(z.string()),
types: z.array(z.string()),
}),
workflow_dispatch: z.unknown(),
}),
permissions: z.record(z.string(), z.string()),
jobs: z.object({
readiness: z.object({
permissions: z.record(z.string(), z.string()),
steps: z.array(
z.object({
uses: z.string(),
with: z.object({
ref: z.string().optional(),
"persist-credentials": z.boolean().optional(),
script: z.string().optional(),
}),
}),
),
}),
}),
})
.parse(
parse(
await readFile(
new URL(
"../../../.github/workflows/pr-readiness.yml",
import.meta.url,
),
"utf8",
),
),
);
it("executes the production workflow step with trusted code and status-only write authority", async () => {
const workflow = await readWorkflow();
expect(workflow.on.pull_request_target.branches).toEqual([
"main",
"release/*",
]);
expect(workflow.on.pull_request_target.types).toEqual(
expect.arrayContaining([
"opened",
"edited",
"synchronize",
"reopened",
"ready_for_review",
"converted_to_draft",
]),
);
expect(workflow.permissions).toEqual({});
expect(workflow.jobs.readiness.permissions).toEqual({
contents: "read",
"pull-requests": "read",
statuses: "write",
});
const [checkout, check] = workflow.jobs.readiness.steps;
expect(checkout?.with).toEqual({
ref: "${{ github.event.repository.default_branch }}",
"persist-credentials": false,
});
expect(check?.uses).toMatch(/^actions\/github-script@[a-f\d]{40}$/u);
const input = fixture();
const root = fileURLToPath(new URL("../../../", import.meta.url));
const code = `
import { createRequire } from "node:module";
const require = createRequire(${JSON.stringify(new URL("../../../package.json", import.meta.url).href)});
const pull = ${JSON.stringify(input.pull)};
const statuses = [];
const github = {
rest: {
pulls: { get: async () => ({ data: pull }) },
repos: {
createCommitStatus: async status => statuses.push(status),
listPullRequestsAssociatedWithCommit: () => {},
},
},
paginate: async () => [pull],
};
const context = ${JSON.stringify(input.context)};
const core = {
info: () => {},
setFailed: message => { throw new Error(message); },
};
${z.string().parse(check?.with.script)}
console.log(JSON.stringify(statuses));
`;
const { stdout } = await promisify(execFile)(
process.execPath,
["--input-type=module", "-e", code],
{
cwd: root,
env: { ...process.env, GITHUB_WORKSPACE: root },
encoding: "utf8",
timeout: 10_000,
maxBuffer: 1024 * 1024,
},
);
const statuses = z
.array(z.object({ state: z.string() }))
.parse(JSON.parse(stdout));
expect(statuses.at(-1)?.state).toBe("success");
});
@@ -12,6 +12,7 @@ import { createEvidence, parseEvidence } from "../../src/domain/evidence.js";
import type { JsonValue } from "../../src/domain/jsonValue.js";
import { analysisCliErrorEnvelopeSchema } from "../../src/contracts/errorSchemas.js";
import { toolContract } from "../../src/contracts/toolContracts.js";
import { javascriptApplicationAnalysisResultSchema } from "../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { javaScriptSemanticTraceResultSchema } from "../../src/domain/javascript/javascriptSemanticTraceSchemas.js";
import { resolveJavaScriptSemanticEvidence } from "../../src/domain/javascript/javascriptSemanticGraph.js";
@@ -124,12 +125,88 @@ it("reports invalid nested semantic graph Evidence as caller input through CLI a
}
});
it("excludes local exports from writes-only traces through CLI and MCP", async () => {
const root = await createTestTempDirectory("rea-semantic-export-trace-");
await mkdir(join(root, "src"));
await writeFile(
join(root, "src", "main.js"),
[
"const answer = 42;",
"export { answer, answer as result };",
"let mutable = 0;",
"mutable = 1;",
"export { mutable };",
].join("\n"),
);
const cli = createCli({});
const analyzed = await serveCli(cli, [
"analyze-javascript-application",
root,
"--json",
]);
expect(analyzed.exitCode).toBe(0);
const application = parseEvidence(JSON.parse(analyzed.stdout));
const analysisResult = javascriptApplicationAnalysisResultSchema.parse(
application.normalized_result,
);
const bindingId = (label: string): string => {
const binding = analysisResult.semantic_graph.nodes.find(
(node) => node.kind === "binding" && node.label === label,
);
if (binding === undefined)
throw new Error(`CLI semantic graph must include the ${label} binding`);
return binding.node_id;
};
const harness = await createApplicationMcpHarness();
try {
const exportTrace = await assertSuccessfulTraceParity(
cli,
harness,
application,
{
seed: { kind: "semantic-node", node_id: bindingId("answer") },
direction: "backward-provenance",
allowed_relations: ["writes"],
},
);
expect(exportTrace.summary).toMatchObject({
total_seed_matches: 1,
traversed_relations: 0,
});
expect(exportTrace.relations).toEqual([]);
const assignmentTrace = await assertSuccessfulTraceParity(
cli,
harness,
application,
{
seed: { kind: "semantic-node", node_id: bindingId("mutable") },
direction: "backward-provenance",
allowed_relations: ["writes"],
},
);
expect(assignmentTrace.summary).toMatchObject({
total_seed_matches: 1,
traversed_relations: 1,
});
expect(assignmentTrace.relations).toHaveLength(1);
} finally {
await harness.close();
}
});
const assertSuccessfulTraceParity = async (
cli: ReturnType<typeof createCli>,
harness: Awaited<ReturnType<typeof createApplicationMcpHarness>>,
application: ReturnType<typeof parseEvidence>,
query: { readonly seed: unknown; readonly direction: string },
): Promise<void> => {
query: {
readonly seed: unknown;
readonly direction: string;
readonly allowed_relations?: readonly string[];
},
): Promise<ReturnType<typeof javaScriptSemanticTraceResultSchema.parse>> => {
const arguments_ = { application, query };
const cliResponse = await serveCli(cli, [
"trace-javascript-semantics",
@@ -162,6 +239,7 @@ const assertSuccessfulTraceParity = async (
).outputSchema.parse(mcpResponse.structuredContent);
expect(mcpEvidence.normalized_result).toEqual(cliResult);
expect(mcpEvidence).toEqual(cliEvidence);
return cliResult;
};
const assertTraceContextsResolve = (
@@ -2,8 +2,6 @@ import { afterEach, describe, expect, it } from "vitest";
import type { AnalysisOperationPort } from "../../../src/application/AnalysisProvider.js";
import { EnhancedTools } from "../../../src/application/EnhancedTools.js";
import { AnalysisOutputError } from "../../../src/domain/analysisErrorCore.js";
import { err } from "../../../src/domain/result.js";
import {
closeEnhancedToolResources,
@@ -16,43 +14,15 @@ import { observed as ok } from "../../fixtures/analysisExecution.js";
afterEach(closeEnhancedToolResources);
describe("enhanced MCP tools", () => {
it("returns typed graph failures and stable unresolved-name results", async () => {
it("returns stable unresolved-name results", async () => {
const tools = new EnhancedTools({
execute: (name) =>
name === "procedure_address"
? Promise.resolve(ok("0x1"))
: name === "procedure_callees"
? Promise.resolve(err(new AnalysisOutputError(name, "failed")))
: Promise.resolve(ok(inventory({}))),
execute: () => Promise.resolve(ok(inventory({}))),
});
const graph = await tools.execute("get_call_graph", {
address: "0x1",
direction: "forward",
});
const unresolved = await tools.execute("find_xrefs_to_name", {
name: "missing",
});
expect(graph).toMatchObject({
ok: true,
value: {
graph: {
"0": [
{
address: "0x1",
status: "error",
error: {
category: "execution_failure",
message:
"Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.",
},
},
],
},
limitations: [],
},
});
expect(unresolved).toEqual({
ok: true,
value: {
@@ -2,7 +2,6 @@ import { afterEach, describe, expect, it } from "vitest";
import type { AnalysisOperationPort } from "../../../src/application/AnalysisProvider.js";
import { createAnalysisExecution } from "../../../src/application/AnalysisProvider.js";
import { EnhancedTools } from "../../../src/application/EnhancedTools.js";
import {
AnalysisCapabilityUnavailableError,
AnalysisCancelledError,
@@ -11,7 +10,12 @@ import {
import type { RelationshipCoverage } from "../../../src/domain/relationshipCoverage.js";
import { err, ok as resultOk } from "../../../src/domain/result.js";
import { closeEnhancedToolResources, connect } from "./enhancedToolsHarness.js";
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
import {
closeEnhancedToolResources,
connect,
jsonResult,
} from "./enhancedToolsHarness.js";
import { observed as ok } from "../../fixtures/analysisExecution.js";
afterEach(closeEnhancedToolResources);
@@ -312,6 +316,98 @@ describe("trace_call_path MCP outcome semantics", () => {
});
});
describe("enhanced MCP batch cancellation", () => {
it.each([
{
stage: "selector resolution",
cancelAt: "procedure_address",
expectedCalls: ["procedure_address:0x1"],
operation: "procedure_address",
},
{
stage: "entry-label lookup",
cancelAt: "address_name",
expectedCalls: ["procedure_address:0x1", "address_name:0x1"],
operation: "address_name",
},
{
stage: "pseudocode read",
cancelAt: "procedure_pseudo_code",
expectedCalls: [
"procedure_address:0x1",
"address_name:0x1",
"procedure_pseudo_code:0x1",
],
operation: "procedure_pseudo_code",
},
] as const)(
"returns typed cancellation from $stage without dispatching a later stage",
async ({ cancelAt, expectedCalls, operation }) => {
const calls: string[] = [];
const signals: boolean[] = [];
const client = await connect({
execute: async (name, arguments_, options) => {
const selector = String(arguments_.procedure ?? arguments_.address);
calls.push(`${name}:${selector}`);
signals.push(options?.signal?.aborted ?? false);
if (name === cancelAt)
return err(new AnalysisCancelledError(operation));
if (name === "procedure_address") return ok(selector);
if (name === "address_name") return ok("fixture_name");
if (name === "procedure_pseudo_code") return ok("return 0;");
throw new Error(`Unexpected operation: ${name}`);
},
});
const response = await client.callTool({
name: "batch_decompile",
arguments: { addresses: ["0x1"] },
});
expect(response.isError).toBe(true);
expect(response.structuredContent).toBeUndefined();
expect(parseMcpToolError(response)).toMatchObject({
error: { code: "cancelled", details: { operation } },
});
expect(calls).toEqual(expectedCalls);
expect(signals).toEqual(expectedCalls.map(() => false));
},
);
it("stops later batch stages across the input when one entry lookup is cancelled", async () => {
const calls: string[] = [];
const client = await connect({
execute: async (name, arguments_) => {
const selector = String(arguments_.procedure ?? arguments_.address);
calls.push(`${name}:${selector}`);
if (name === "procedure_address") return ok(selector);
if (name === "address_name" && selector === "0x1")
return err(new AnalysisCancelledError(name));
if (name === "address_name") return ok("fixture_name");
if (name === "procedure_pseudo_code") return ok("return 0;");
throw new Error(`Unexpected operation: ${name}`);
},
});
const response = await client.callTool({
name: "batch_decompile",
arguments: { addresses: ["0x1", "0x2"] },
});
expect(response.isError).toBe(true);
expect(response.structuredContent).toBeUndefined();
expect(parseMcpToolError(response)).toMatchObject({
error: { code: "cancelled", details: { operation: "address_name" } },
});
expect(calls).toEqual([
"procedure_address:0x1",
"procedure_address:0x2",
"address_name:0x1",
"address_name:0x2",
]);
});
});
describe("enhanced MCP batches", () => {
it("returns every batch item in caller order with its own result", async () => {
const addresses = Array.from(
@@ -358,61 +454,143 @@ describe("enhanced MCP batches", () => {
});
it("returns ordered typed batch failures and zero counts for empty input", async () => {
const tools = new EnhancedTools({
execute: (name, arguments_) =>
const client = await connect({
execute: async (name, arguments_) =>
name === "procedure_address"
? Promise.resolve(ok(String(arguments_.procedure)))
? ok(String(arguments_.procedure))
: name === "address_name"
? Promise.resolve(ok(null))
? ok(null)
: arguments_.procedure === "0x2"
? Promise.resolve(
err(new AnalysisOutputError("decompile", "failed")),
)
: Promise.resolve(ok("pseudo")),
? err(new AnalysisOutputError("decompile", "failed"))
: ok("pseudo"),
});
const result = await tools.execute("batch_decompile", {
addresses: ["0x1", "0x2"],
const result = await client.callTool({
name: "batch_decompile",
arguments: { addresses: ["0x1", "0x2"] },
});
const empty = await client.callTool({
name: "batch_decompile",
arguments: { addresses: [] },
});
const empty = await tools.execute("batch_decompile", { addresses: [] });
expect(result).toEqual({
ok: true,
value: {
items: [
{
address: "0x1",
procedure: { status: "resolved", address: "0x1", name: null },
status: "ok",
pseudocode: "pseudo",
},
{
address: "0x2",
procedure: { status: "resolved", address: "0x2", name: null },
status: "error",
error: {
code: "unreadable_output",
category: "execution_failure",
details: { operation: "decompile", reason: "failed" },
message:
expect(result.isError).not.toBe(true);
expect(jsonResult(result)).toEqual({
items: [
{
address: "0x1",
procedure: { status: "resolved", address: "0x1", name: null },
status: "ok",
pseudocode: "pseudo",
},
{
address: "0x2",
procedure: { status: "resolved", address: "0x2", name: null },
status: "error",
error: {
code: "unreadable_output",
category: "execution_failure",
details: { operation: "decompile", reason: "failed" },
message:
"Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.",
retryable: false,
remediation: {
action:
"Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.",
retryable: false,
remediation: {
action:
"Analysis returned an unreadable result. Run `rea doctor` and review the reported details; repeating the unchanged request is not expected to succeed.",
},
},
},
],
total: 2,
succeeded: 1,
failed: 1,
},
],
total: 2,
succeeded: 1,
failed: 1,
});
expect(empty.isError).not.toBe(true);
expect(jsonResult(empty)).toEqual({
items: [],
total: 0,
succeeded: 0,
failed: 0,
});
});
});
describe("enhanced MCP graph cancellation", () => {
it("returns typed relationship cancellation without expanding queued children", async () => {
const calls: string[] = [];
const signals: boolean[] = [];
const client = await connect({
execute: async (name, arguments_, options) => {
const selector = String(arguments_.procedure ?? arguments_.address);
calls.push(`${name}:${selector}`);
signals.push(options?.signal?.aborted ?? false);
if (name === "procedure_address") return relationshipExecution("0x1");
if (name !== "procedure_callees")
throw new Error(`Unexpected operation: ${name}`);
if (selector === "0x1") return relationshipExecution(["0x2", "0x3"]);
if (selector === "0x2") return err(new AnalysisCancelledError(name));
throw new Error("queued child 0x3 must not be expanded");
},
});
expect(empty).toEqual({
ok: true,
value: { items: [], total: 0, succeeded: 0, failed: 0 },
const response = await client.callTool({
name: "get_call_graph",
arguments: { address: "0x1", direction: "forward" },
});
expect(response.isError).toBe(true);
expect(response.structuredContent).toBeUndefined();
expect(parseMcpToolError(response)).toMatchObject({
error: { code: "cancelled", details: { operation: "procedure_callees" } },
});
expect(calls).toEqual([
"procedure_address:0x1",
"procedure_callees:0x1",
"procedure_callees:0x2",
]);
expect(signals).toEqual([false, false, false]);
});
it("keeps ordinary provider failures as partial nodes and continues queued work", async () => {
const calls: string[] = [];
const client = await connect({
execute: async (name, arguments_) => {
const selector = String(arguments_.procedure ?? arguments_.address);
calls.push(`${name}:${selector}`);
if (name === "procedure_address") return relationshipExecution("0x1");
if (name !== "procedure_callees")
throw new Error(`Unexpected operation: ${name}`);
if (selector === "0x1") return relationshipExecution(["0x2", "0x3"]);
if (selector === "0x2")
return err(new AnalysisOutputError(name, "ordinary fixture failure"));
return relationshipExecution([]);
},
});
const response = await client.callTool({
name: "get_call_graph",
arguments: { address: "0x1", direction: "forward" },
});
expect(response.isError).not.toBe(true);
expect(jsonResult(response)).toMatchObject({
graph: {
"1": [
{
address: "0x2",
status: "error",
error: { code: "unreadable_output" },
},
{ address: "0x3", status: "ok", calls: [] },
],
},
});
expect(calls).toEqual([
"procedure_address:0x1",
"procedure_callees:0x1",
"procedure_callees:0x2",
"procedure_callees:0x3",
]);
});
});
@@ -10,7 +10,9 @@ import { z } from "zod";
import { parseEvidence } from "../../../src/domain/evidence.js";
import { javascriptApplicationAnalysisResultSchema } from "../../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { writeJavaScriptSourceSelectionFixture } from "../../fixtures/javascriptSourceSelection.js";
import { createStrippedAsarAddon } from "../../fixtures/strippedAsarAddon.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { writeFixtureFiles } from "../../support/javascriptApplicationFixture.js";
const createSelectionClient = () => {
const client = new Client({
@@ -30,6 +32,79 @@ const createSelectionClient = () => {
return { client, transport };
};
it("resolves ESM whitespace specifiers through the public application tool", async () => {
const root = await createTestTempDirectory("rea-esm-whitespace-mcp-");
const rawSpecifier = "./feature\tname.js";
const encodedSpecifier = "./feature%09name.js";
await writeFixtureFiles(root, {
"main.mjs": `import ${JSON.stringify(rawSpecifier)};\nimport ${JSON.stringify(encodedSpecifier)};\n`,
"featurename.js": "export const plain = 1;\n",
"feature\tname.js": "export const value = 1;\n",
});
const { client, transport } = createSelectionClient();
try {
await client.connect(transport);
const output = await client.callTool({
name: "analyze_javascript_application",
arguments: { input_path: root },
});
expect(output.isError, JSON.stringify(output.content)).not.toBe(true);
const result = javascriptApplicationAnalysisResultSchema.parse(
parseEvidence(output.structuredContent).normalized_result,
);
expect(result.graph.edges).toEqual(
expect.arrayContaining([
expect.objectContaining({
properties: expect.objectContaining({
specifier: rawSpecifier,
resolved_path: "featurename.js",
}),
}),
expect.objectContaining({
properties: expect.objectContaining({
specifier: encodedSpecifier,
resolved_path: "feature\tname.js",
}),
}),
]),
);
} finally {
await client.close();
await transport.close();
}
}, 60_000);
it("retains missing ASAR native-addon inventory through the public application tool", async () => {
const { archive, addon } = await createStrippedAsarAddon();
await rm(join(`${archive}.unpacked`, addon));
const { client, transport } = createSelectionClient();
try {
await client.connect(transport);
const output = await client.callTool({
name: "analyze_javascript_application",
arguments: { input_path: archive },
});
expect(output.isError, JSON.stringify(output.content)).not.toBe(true);
const result = javascriptApplicationAnalysisResultSchema.parse(
parseEvidence(output.structuredContent).normalized_result,
);
expect(result.graph.coverage.status).toBe("partial");
expect(
result.graph.nodes.some(
(node) =>
node.kind === "native-addon" &&
node.observations.some(
({ label, properties }) =>
label === addon && properties.availability === "unavailable",
),
),
).toBe(true);
} finally {
await client.close();
await transport.close();
}
}, 60_000);
it("advertises exact source selection and retains its scope through summary, Evidence retrieval and invalid-input recovery", async () => {
const root = await createTestTempDirectory("rea-js-selected-mcp-");
await writeJavaScriptSourceSelectionFixture(root);
@@ -20,8 +20,12 @@ const UNTRUSTED_TARGET_CONTENT = {
provenance: { untrusted_content: true, source: "analysis_target" },
};
const failureWithName = (nameBytes: number, targetContent = false) =>
delivery.toErrorToolResult(
const failureWithName = (
nameBytes: number,
targetContent = false,
selectedDelivery = delivery,
) =>
selectedDelivery.toErrorToolResult(
new AnalysisInputError("procedure_address", undefined, [
{
path: ["procedure"],
@@ -260,10 +264,15 @@ it("keeps the untrusted target-content marker on an oversized error's replacemen
// The delivered error wire omits only the private structured carrier; its
// `isError` flag and any provenance marker count against the result budget.
const deliveredErrorBytes = (nameBytes: number, targetContent: boolean) => {
const deliveredErrorBytes = (
nameBytes: number,
targetContent: boolean,
selectedDelivery = delivery,
) => {
const { structuredContent: _carrier, ...wire } = failureWithName(
nameBytes,
targetContent,
selectedDelivery,
);
return Buffer.byteLength(JSON.stringify(wire));
};
@@ -271,26 +280,28 @@ const deliveredErrorBytes = (nameBytes: number, targetContent: boolean) => {
it.each([false, true])(
"applies the result budget to the complete delivered error (target content: %s)",
async (targetContent) => {
const boundaryDelivery = new ToolResultDelivery(4096);
const fittingName =
delivery.resultBudgetBytes - deliveredErrorBytes(0, targetContent);
expect(deliveredErrorBytes(fittingName, targetContent)).toBe(
delivery.resultBudgetBytes,
);
boundaryDelivery.resultBudgetBytes -
deliveredErrorBytes(0, targetContent, boundaryDelivery);
expect(
deliveredErrorBytes(fittingName, targetContent, boundaryDelivery),
).toBe(boundaryDelivery.resultBudgetBytes);
const toolName = targetContent ? "procedure_pseudo_code" : "record_unknown";
const server = new EvidenceMcpServer(
{ name: "error-budget-test", version: "1" },
{ capabilities: {} },
undefined,
delivery,
boundaryDelivery,
);
server.registerTool(toolName, { inputSchema: z.object({}) }, async () =>
failureWithName(fittingName, targetContent, boundaryDelivery),
);
server.registerTool(
"fitting_failure",
targetContent ? "inspect_artifact" : "export_evidence_bundle",
{ inputSchema: z.object({}) },
async () => failureWithName(fittingName, targetContent),
);
server.registerTool(
"one_byte_over_failure",
{ inputSchema: z.object({}) },
async () => failureWithName(fittingName + 1, targetContent),
async () =>
failureWithName(fittingName + 1, targetContent, boundaryDelivery),
);
const client = new Client({ name: "error-budget-client", version: "1" });
const [clientTransport, serverTransport] =
@@ -299,15 +310,19 @@ it.each([false, true])(
await server.connect(serverTransport);
await client.connect(clientTransport);
const fitting = await client.callTool({
name: "fitting_failure",
name: toolName,
arguments: {},
});
expect(fitting.isError).toBe(true);
expect(fitting._meta).toEqual(
targetContent ? UNTRUSTED_TARGET_CONTENT : undefined,
);
expect(parseMcpToolError(fitting)).toEqual(
failureWithName(fittingName, targetContent).structuredContent,
failureWithName(fittingName, targetContent, boundaryDelivery)
.structuredContent,
);
const over = await client.callTool({
name: "one_byte_over_failure",
name: targetContent ? "inspect_artifact" : "export_evidence_bundle",
arguments: {},
});
expect(over.isError).toBe(true);
+65 -1
View File
@@ -19,6 +19,8 @@ import type {
ProviderAvailability,
} from "../../../src/application/AnalysisProvider.js";
import type { AndroidAnalysisPort } from "../../../src/application/android/AndroidAnalysisPort.js";
import type { AdbDeviceAnalysisPort } from "../../../src/application/adb/AdbDeviceAnalysisPort.js";
import type { ApktoolResourceAnalysisPort } from "../../../src/application/apktool/ApktoolResourceAnalysisPort.js";
import { observed as ok } from "../../fixtures/analysisExecution.js";
import { createServer } from "../../../src/server/createServer.js";
import { silentLogger } from "../../../src/logger.js";
@@ -26,7 +28,10 @@ import { createAnalysisProfile } from "../../../src/domain/analysisProfile.js";
import { MAX_JSON_DEPTH } from "../../../src/domain/jsonValue.js";
import { parseAnalysisSnapshot } from "../../../src/domain/analysisSnapshot.js";
import { INVESTIGATION_EXAMPLES } from "../../../src/contracts/investigationExamples.js";
import { expandProviderCapabilities } from "../../../src/contracts/toolOutputSchemaPrimitives.js";
import {
expandProviderCapabilities,
toolAvailabilityInventory,
} from "../../../src/contracts/toolOutputSchemaPrimitives.js";
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { ok as resultOk } from "../../../src/domain/result.js";
import {
@@ -153,6 +158,65 @@ describe("MCP snapshot lifecycle ordering", () => {
}, 10_000);
});
describe("public provider availability", () => {
it("reports ADB and Apktool host probes in the public session inventory", async () => {
const closed: string[] = [];
const session = createTestBinarySession(provider(closed));
const available: ProviderAvailability = {
status: "available",
code: null,
reason: null,
diagnostics: { version: "1.0.41" },
};
const unavailable: ProviderAvailability = {
status: "unavailable",
code: "unsupported_version",
reason: "Use Apktool 2.7.0 or newer.",
diagnostics: { minimum_version: "2.7.0" },
};
const adbDeviceAnalysis: AdbDeviceAnalysisPort = {
inspectAvailability: async () => available,
close: async () => {},
execute: async () => ok(null),
};
const apktoolAnalysis: ApktoolResourceAnalysisPort = {
inspectAvailability: async () => unavailable,
close: async () => {},
execute: async () => ok(null),
};
const server = createServer(
{ kind: "session", session },
{ adbDeviceAnalysis, apktoolAnalysis, logger: silentLogger },
);
const mcp = new Client({
name: "provider-host-readiness",
version: "1.0.0",
});
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
resources.push(mcp, server);
await server.connect(serverTransport);
await mcp.connect(clientTransport);
const status = structured(
await mcp.callTool({ name: "binary_session", arguments: {} }),
);
const availability = toolAvailabilityInventory.parse(
(status.result as { tool_availability: unknown }).tool_availability,
);
expect(
availability.groups.find(({ names }) =>
names.includes("inspect_adb_client"),
),
).toMatchObject({ available: true, reason: "available" });
expect(
availability.groups.find(({ names }) =>
names.includes("inspect_apktool_client"),
),
).toMatchObject({ available: false, reason: "provider_unavailable" });
});
});
describe("target-free MCP lifecycle", () => {
it("takes the session snapshot after Android readiness inspection completes", async () => {
directory = await createTestTempDirectory("rea-mcp-status-readiness-");
@@ -70,6 +70,98 @@ it("rejects oversized initial and resize terminal buffers through MCP before lau
await client.ping();
});
itWithCaptureCapability(
"bounds normalized PTY text and preserves equal-size output through MCP",
async () => {
const root = await createTestTempDirectory(
"rea-process-normalized-budget-",
);
const session = createTestBinarySession(createCacheProvider([]));
const server = createServer(
{ kind: "session", session },
{ logger: silentLogger },
);
const client = new Client({
name: "process-normalized-budget",
version: "1",
});
resources.push(client, server, session);
const [clientTransport, serverTransport] =
InMemoryTransport.createLinkedPair();
await server.connect(serverTransport);
await client.connect(clientTransport);
const capture = async (replacement: string) => {
const result = await client.callTool({
name: "capture_process_scenario",
arguments: {
executable: process.execPath,
working_directory: root,
arguments: ["-e", 'process.stdout.write("a".repeat(100) + "\\n")'],
limits: { output_bytes: 1_000 },
normalization: {
paths: false,
pids: false,
ports: false,
patterns: [{ pattern: "a", replacement }],
},
settle_ms: 50,
},
});
expect(result.isError, JSON.stringify(result.content)).not.toBe(true);
return toolContract("capture_process_scenario").outputSchema.parse(
result.structuredContent,
).normalized_result;
};
const expanded = await capture("x".repeat(500));
expect(expanded.frames).toHaveLength(0);
expect(expanded.truncated).toBe(true);
expect(expanded.truncation_details.normalized_terminal).toMatchObject({
budget_bytes: 1_000,
retained_bytes: 0,
omitted_frames: expect.any(Number),
});
expect(
expanded.truncation_details.normalized_terminal?.omitted_frames,
).toBeGreaterThan(0);
expect(expanded.truncation_details.raw_terminal).toMatchObject({
budget_bytes: 1_000,
observed_bytes: expect.any(Number),
retained_bytes: 0,
observed_frames: expect.any(Number),
retained_frames: 0,
});
expect(
expanded.limitations.some((limitation) =>
limitation.includes(
"final normalized sizes for the latter omissions are unknown",
),
),
).toBe(true);
expect(expanded.cleanup).toMatchObject({
owned_process_group: "verified",
temporary_root: "removed",
});
const equalSize = await capture("b");
expect(equalSize.truncated).toBe(false);
expect(equalSize.frames.map(({ data }) => data).join("")).toBe(
`${"b".repeat(100)}\r\n`,
);
expect(equalSize.truncation_details.normalized_terminal).toMatchObject({
budget_bytes: 1_000,
retained_bytes: Buffer.byteLength(`${"b".repeat(100)}\r\n`),
omitted_frames: 0,
});
expect(equalSize.cleanup).toMatchObject({
owned_process_group: "verified",
temporary_root: "removed",
});
},
30_000,
);
itWithCaptureCapability(
"records process residuals linked to capture Evidence",
async () => {