Compare commits

..
Author SHA1 Message Date
gensokyo-dev 6381aa0b1d fix(ghidra): scope rejected-input diagnostics to their request (#7140)
Scope expected Ghidra input failures to the active request after the authenticated startup handshake. Preserve operation output, identity and log coordinates while retaining cumulative output with its session owner; lifecycle and unexpected failures keep full diagnostics.

Share remote input-code classification with domain error projection. Verify startup rejection, healthy continuation, cleanup, unknown errors, OOM diagnostics and real CLI/MCP workflows. The measured unknown-symbol reply falls from 13,170 to 2,209 bytes with unchanged successful facts and Evidence IDs.
2026-10-25 22:37:32 +08:00
gensokyo-dev f69c52a1cf feat(native): page retained load-image observations (#7134)
Add optional native load-image summary delivery and retained selected-row pages through shared CLI/MCP analysis views. Preserve source verification, provenance, unavailable collections, exact row identity and opt-in byte/check payloads.

Preserve direct caller paths during shared CLI summary publication and cover raw/DOL projection contracts through the actual SDK. Real Ghidra ELF CLI/MCP captures, retained and file-backed pages, focused regressions and static checks passed; the PR records existing main schema failures and the JavaScript resource-recovery CI uncertainty.
2026-10-25 21:50:42 +08:00
43 changed files with 3154 additions and 229 deletions

No files matched your search

@@ -430,6 +430,14 @@ loaded bytes, file mappings, relocations and the entry against the immutable
snapshot. Keep `mismatch` and `unsupported` explicit. Verified import covers the
reported static image, not runtime DOS or PC-98 hardware behavior.
For a large load-image report, select `detail: "summary"` and query its
`parent_evidence_id` with `inspect_analysis_view` kind `native-load-image`.
Choose a collection such as `relocations` or `mappings`, select only needed
fields, and page with offset/limit. Original indices preserve same-address
alternatives. Null collections remain unrecorded or inapplicable; paging does
not upgrade unsupported verification. CLI requires `--evidence_output` for
summary delivery so complete source Evidence remains accessible.
For Ghidra decompilation, `procedure_pseudo_code` also accepts `procedure: "file-offset:0x..."`; CLI `rea decompile <path> --file-offset=0x... --provider ghidra` resolves the source byte offset internally. An unresolved 0x-prefixed numeric procedure selector also falls back to a unique file mapping. Existing symbols and analysis addresses win; multiple mappings fail as ambiguous. Preserve the returned file-offset and address locations and raw selection observations. ELF virtual addresses are distinct from file offsets; never assume an arbitrary offset plus image base is an analysis address.
Complete function body ranges use inclusive ends and can contain gaps. Do not
+48
View File
@@ -30,6 +30,17 @@ signal. Confirm resource causes from process output or host/container records;
operation-scoped JVM `OutOfMemoryError` observations retain their separate resource
classification. Automatic kernel or cgroup OOM attribution is not implemented.
After the authenticated startup handshake, rejected inputs (`invalid_request`,
`not_found`, `ambiguous`, or an invalid function name) use
`diagnostics.diagnostic_scope: "request"` when their active output checkpoint is
available. They preserve the active request's
`failure_output`, target/profile identity and runtime/log coordinates, while
omitting cumulative launcher stdout/stderr and launch-support output. Startup,
process, protocol and other remote failures retain full session diagnostics.
The Ghidra client still owns its bounded cumulative output; this projection does
not clear it or alter session readiness. Log paths refer to the owned ephemeral
runtime and are removed by normal cleanup.
Provider selection and profile identity follow
[MCP contracts](mcp-contracts.md#identity-and-discovery). One deep
provider stays bound to the target; runtime failure never selects another
@@ -160,6 +171,43 @@ have no file backing. ELF mappings are observed
without independent format-specific load-image verification, as the result's
status and limitations state.
For large relocation inventories, select MCP `inspect_native_load_image` with
`detail: "summary"`. It returns verification state, observed identity and counts,
while retaining complete Evidence. Use `normalized_result.parent_evidence_id`
with `inspect_analysis_view` and a view such as:
```json
{
"kind": "native-load-image",
"collection": "relocations",
"offset": 0,
"limit": 32,
"fields": ["address", "status", "type"]
}
```
Collections are `source_files`, `mappings`, `relocations`, `entry_points`,
`entry_context`, `checks` and DOL `sections`. Optional
`indices` select original row indices before paging; empty indices select none.
Fields are collection-specific. Relocation defaults omit byte payloads; check defaults omit potentially large `expected`/`observed` inventories. Request those fields explicitly when needed. Other collections default to all fields. Metadata recovery remains counted and retained in complete parent Evidence.
Parent Evidence, collection and `source_index` identify a row, including same-address
relocation alternatives. Null counts or rows mean unrecorded or inapplicable;
they never establish an empty observation. Exhausting a page does not upgrade
unsupported verification or runtime coverage. Pages query immutable retained
facts without reopening the target or repeating Ghidra work.
Each page carries the source's structured `verification_status` independently
of its page coverage.
CLI summary delivery requires a complete Evidence bundle:
```sh
rea inspect-native-load-image <path> --provider ghidra --detail summary --evidence_output /new/load-image.json --json
```
Read selected pages later through `inspect-analysis-view` using the returned
parent ID and an `evidence-file` source pointing to that bundle. Complete
delivery remains the default in both adapters.
An unsuccessful numeric procedure lookup reports the current Ghidra image base
and gives the CLI commands above alongside the corresponding MCP operations.
Repeat any target-format, architecture, and entry-point options used to open the
+22 -4
View File
@@ -1220,16 +1220,34 @@ after SDK conversion and check advertised validation and actual calls.
Individual model APIs can impose additional nesting limits; complete producer
captures can exceed ten structural levels.
Claude Desktop drops a local connector tool, with a generic "problem with that
connector" warning, when its input schema serialized as compact JSON exceeds
64 KiB ([#4081](https://github.com/morluto/rea/issues/4081)). Every advertised
input schema therefore fits 65,536 UTF-8 bytes, including its examples. A
self-contained schema that would exceed the limit is fitted in two steps, and
its root description says it is summarized. First, a property whose schema
repeats an earlier sibling's, such as `after` beside `before`, is advertised as
its types and description plus "Same schema as `before`." Second, schemas below
the deepest instance level that still fits are advertised as their types, size
constraints and descriptions, unless that summary would be longer than the
schema itself. Combinator and conditional branches count as the
level of the instance they describe. Both steps only widen the advertisement:
canonical inputs and examples stay accepted, required names and nested
constraints below the cut are not advertised, and the canonical contract still
validates every call. Schemas within the limit are advertised unchanged. The
compact profile starts from the complete projection, not from the fitted one.
`search_bytes` advertises its pattern and mask as descriptive strings. Its
shared canonical validator scans byte pairs, whitespace, wildcards and aligned
masks without a repeating regular expression or temporary byte arrays. The
server rejects malformed or unconstrained patterns before provider dispatch;
the advertised strings alone do not express those rules.
Boolean and empty-string literal constraints in input schemas are relaxed only
in this advertisement. Boolean `const` becomes `type: boolean`; empty-string
`const` or an enum containing an empty string is widened without dropping the
accepted empty value. Defaults and descriptions remain intact.
Boolean, numeric and empty-string literal constraints in input schemas are
relaxed only in this advertisement. Boolean `const` becomes `type: boolean`;
numeric `const` keeps its numeric type; empty-string `const`, an enum containing
an empty string, and an enum containing a number are widened without dropping
an accepted value. Defaults and descriptions remain intact.
An affected `not` is omitted, conditional predicates and their branches are
omitted together, pure boolean-literal `oneOf` branches collapse to a boolean,
and other affected `oneOf` branches are presented as `anyOf` to retain branch
@@ -0,0 +1,89 @@
# Native load-image delivery verification
`native-load-image-views.mjs` runs the compiled production stdio MCP server with
a caller-selected Ghidra and JDK. It compares complete delivery with a summary
plus selected relocation pages, then exports and authenticates the complete
parent. Every call uses its actual advertised SDK contract. Both cold and warm
reads run in one fresh server; OS file caches are not declared cold.
Build each selected runtime with the pinned toolchain. Use absolute paths or
paths relative to the verifier's launch directory, and new output directories:
```sh
node experiments/analysis-efficiency/native-load-image-views.mjs \
--runtime /path/to/baseline --mode complete \
--target /usr/lib/x86_64-linux-gnu/libcrypto.so.3 \
--ghidra /path/to/ghidra_12.1.4_PUBLIC --java /path/to/jdk-21 \
--output /new/baseline-report
node experiments/analysis-efficiency/native-load-image-views.mjs \
--runtime /path/to/feature --mode summary \
--target /usr/lib/x86_64-linux-gnu/libcrypto.so.3 \
--ghidra /path/to/ghidra_12.1.4_PUBLIC --java /path/to/jdk-21 \
--output /new/summary-report --control /new/baseline-report
```
The question is the verification state, all collection counts, and the first
four relocation addresses/statuses. The summary path additionally checks the
last relocation's recorded bytes and unavailable independent verification
checks. It never treats ELF observations as verified. Complete delivery can
exceed the MCP response limit: the baseline then follows the reported retained
reference, exports its source and selects facts locally. That recovery is an
extra workflow, not a successful complete inline MCP reply. Include its calls
and local reads when comparing completed tasks.
The independent GNU `readelf -Wr` oracle checks that every file-table relocation
coordinate survives, including multiplicity. `readelf -Wl` supplies the original
PT_LOAD base independently of Ghidra's chosen import base. Producer order can
differ. Additional provider observations are retained and reported separately;
this oracle does not independently verify their origin. Full observation
digests and selected facts must agree with the control run.
The target SHA-256, native oracle output, raw catalog, arguments/replies,
complete Evidence and phase durations stay in the output directory. Measure
whole-process-family elapsed time, CPU and peak memory externally, with bounded
CPU/heap and a memory stop condition. The child requests a 1.5 GiB Ghidra heap
and a 3 GiB Node heap; neither is a whole-family RSS bound. Estimate tokens over
catalog discovery, arguments, all replies and local recovery facts using a named
tokenizer. Raw serialized schema estimates are not actual model usage; protocol
framing, prompts and client rendering remain unmeasured.
The shared receipt finalizer closes both transports, removes its owned workspace
after successful cleanup, and preserves cleanup/publication failures in the
report. Its phase elapsed field precedes final receipt publication; external
whole-family measurements must include that finalization.
## Recorded Linux x64 verification
One fresh cold/warm session per revision used Node 24.22.1, Ghidra 12.1.4,
two CPUs and the system `libcrypto.so.3` with SHA-256
`76dd3d93e5ee48950a92a58d59b94de8143847f91a80d9682c938767b991577d`.
Baseline `42382651b` has the same tree as main `fc43e285e`; feature `7a53a4a7f`.
Both retained the same parent ID and 21,118 observations. GNU readelf declared
21,117 relocation coordinates, all present; one provider-only coordinate at
relative `0x47dc60` remained visible without an independently established origin.
| Whole two-capture harness | Complete + local recovery | Summary + selected reads |
| --------------------------------------- | -----------------------------: | -----------------------: |
| Direct capture reply | Refused at the 10 MiB boundary | 9,787 bytes |
| Complete parent Evidence | 9,114,888 bytes | 9,114,888 bytes |
| Raw serialization, cl100k_base estimate | 912,927 tokens | 936,730 tokens |
| Elapsed | 164.93 s | 169.88 s |
| Process-family CPU | 281.75 s | 293.06 s |
| Sampled process-family RSS peak | 2.01 GiB | 2.30 GiB |
Token estimates include one raw catalog discovery, call arguments/replies and
baseline local recovery facts. Local command text, prompts, protocol framing
and client rendering were not measured; these are not model-usage totals.
The summary harness also checks tail bytes and unavailable checks through
additional RPCs. Neither lower task tokens, faster analysis nor lower memory
is established by this single pair. The benefit is direct bounded MCP access
to facts that previously required an export and local selection. RSS summed
the owned process family every 100 ms; shared pages can be double-counted and
short peaks missed. OS caches were warm.
The real CLI published the same complete parent with summary delivery, then a
separate CLI process read selected first/last rows from that bundle without
configured Ghidra/JDK. That file-page read took 4.83 s, 5.89 CPU seconds and
0.51 GiB sampled family RSS. This verifies historical projection independently
of reopening a native provider. Other hosts, engines and formats were not
verified in this experiment.
@@ -0,0 +1,357 @@
/** Actual Ghidra/MCP load observations with retained-page and readelf oracles. */
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import { createWriteStream } from "node:fs";
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import { finished } from "node:stream/promises";
import { parseArgs, promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
import { finalizeRetainedQueryReceipt } from "./retained-query-finalization.mjs";
const { values } = parseArgs({
options: Object.fromEntries(
["runtime", "target", "output", "ghidra", "java", "mode", "control"].map(
(name) => [name, { type: "string" }],
),
),
});
for (const key of ["runtime", "target", "output", "ghidra", "java"]) {
assert.ok(values[key], `Missing --${key}`);
values[key] = resolve(values[key]);
}
if (values.control) values.control = resolve(values.control);
assert.ok(["complete", "summary"].includes(values.mode));
await mkdir(values.output);
const hash = (value) => createHash("sha256").update(value).digest("hex");
const { parseEvidence } = await import(
pathToFileURL(join(values.runtime, "dist/domain/evidence.js")).href
);
const { stdout: revision } = await promisify(execFile)("git", [
"-C",
values.runtime,
"rev-parse",
"HEAD",
]);
const { stdout: workingState } = await promisify(execFile)("git", [
"-C",
values.runtime,
"status",
"--porcelain",
]);
const targetSha256 = hash(await readFile(values.target));
const { stdout: oracle } = await promisify(execFile)(
"readelf",
["-Wr", values.target],
{ maxBuffer: 64 * 1024 ** 2 },
);
const relocationOffsets = [
...oracle.matchAll(/^([0-9a-f]+)\s+[0-9a-f]+\s+R_\S+/gm),
].map((row) => row[1]);
await writeFile(join(values.output, "readelf-relocations.txt"), oracle);
const { stdout: programHeaders } = await promisify(execFile)(
"readelf",
["-Wl", values.target],
{ maxBuffer: 1024 ** 2 },
);
const loadAddresses = [
...programHeaders.matchAll(/^\s*LOAD\s+0x[0-9a-f]+\s+0x([0-9a-f]+)/gm),
].map((row) => BigInt(`0x${row[1]}`));
assert.ok(loadAddresses.length > 0, "Expected an ELF load mapping");
const lowestLoad = loadAddresses.reduce((left, right) =>
left < right ? left : right,
);
await writeFile(
join(values.output, "readelf-program-headers.txt"),
programHeaders,
);
const isolatedHome = await mkdtemp(join(tmpdir(), "rea-load-image-verifier-"));
const transport = new StdioClientTransport({
command: process.execPath,
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
cwd: values.runtime,
env: {
HOME: isolatedHome,
PATH: process.env.PATH,
NODE_OPTIONS: "--max-old-space-size=3072",
GHIDRA_INSTALL_DIR: values.ghidra,
JAVA_HOME: values.java,
GHIDRA_HEADLESS_MAXMEM: "1536M",
REA_GHIDRA_STARTUP_TIMEOUT_MS: "600000",
REA_MCP_INPUT_SCHEMA_PROFILE: "compact",
REA_LOG_LEVEL: "silent",
},
stderr: "pipe",
});
const serverLog = createWriteStream(join(values.output, "server.log"));
const serverLogFinished = finished(serverLog);
serverLogFinished.catch(() => {});
transport.stderr?.pipe(serverLog);
const client = new Client({ name: "native-load-image-views", version: "1" });
const transcript = [];
const report = {
mode: values.mode,
runtime: values.runtime,
revision: revision.trim(),
dirty: workingState.length > 0,
artifact: { path: values.target, sha256: targetSha256 },
oracle: {
command: ["readelf", "-Wr", values.target],
sha256: hash(oracle),
relocation_count: relocationOffsets.length,
},
question:
"Report retained load-image verification status, collection counts, and the first four relocation addresses/statuses with source linkage.",
calls: [],
runs: [],
};
const call = createRetainedQueryCaller(client, (receipt, event) => {
report.calls.push(receipt);
transcript.push(event);
});
const result = async (name, input, phase = name) => {
const reply = await call(name, input, phase);
assert.notEqual(reply.isError, true, JSON.stringify(reply));
assert.ok(reply.structuredContent);
return reply.structuredContent;
};
const exportedParent = async (id, label) => {
const path = join(values.output, `${label}-evidence.json`);
await result("export_evidence_bundle", { path }, `${label}:export`);
const bundle = JSON.parse(await readFile(path, "utf8"));
const parent = bundle.records.find((record) => record.evidence_id === id);
assert.ok(parent, `Missing retained source ${id}`);
assert.equal(parent.operation, "inspect_native_load_image");
assert.equal(parent.subject.digest.sha256, targetSha256);
assert.equal(parent.provider.id, "ghidra");
return parseEvidence(parent);
};
const selectedRecords = (rows) =>
rows.slice(0, 4).map(({ address, status }) => ({ address, status }));
let opened = false;
const started = performance.now();
try {
await client.connect(transport);
const catalogStart = performance.now();
const catalog = await client.listTools();
assert.equal(catalog.nextCursor, undefined);
await writeFile(join(values.output, "catalog.json"), JSON.stringify(catalog));
report.catalog = {
elapsed_ms: performance.now() - catalogStart,
bytes: Buffer.byteLength(JSON.stringify(catalog)),
sha256: hash(JSON.stringify(catalog)),
};
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
const original = client.callTool.bind(client);
client.callTool = (input, options) =>
original(input, {
...options,
timeout: 600000,
toolDefinition: definitions.get(input.name),
});
await result("open_binary", { path: values.target, provider_id: "ghidra" });
opened = true;
for (const temperature of ["cold", "warm"]) {
const runStart = performance.now();
const reply = await call(
"inspect_native_load_image",
values.mode === "summary" ? { detail: "summary" } : {},
temperature,
);
let parent;
let delivered;
let id;
if (reply.isError) {
assert.equal(values.mode, "complete");
const text = reply.content.find((item) => item.type === "text");
assert.ok(text);
const error = JSON.parse(text.text).error;
assert.equal(error.code, "resource_constraint");
assert.equal(error.details.reported_limits.boundary, "mcp-response");
id = error.details.reported_limits.evidence_reference.evidence_id;
parent = await exportedParent(id, temperature);
delivered = parent.normalized_result;
} else {
assert.ok(reply.structuredContent);
delivered = reply.structuredContent.normalized_result;
id =
values.mode === "summary"
? delivered.parent_evidence_id
: reply.structuredContent.evidence_id;
parent = await exportedParent(id, temperature);
if (values.mode === "complete")
assert.deepEqual(parent, reply.structuredContent);
}
const source = parent.normalized_result;
const observed = source.observations;
// Every file-table coordinate must survive, including repeated coordinates.
// Provider-added observations are retained separately; this oracle does not
// independently establish why the provider created them.
const base = BigInt(observed.image_base.split(":").at(-1));
const counts = new Map();
for (const row of observed.relocations) {
const key = (BigInt(row.address.split(":").at(-1)) - base).toString(16);
counts.set(key, (counts.get(key) ?? 0) + 1);
}
for (const offset of relocationOffsets) {
const key = (BigInt(`0x${offset}`) - lowestLoad).toString(16);
const remaining = counts.get(key) ?? 0;
assert.ok(remaining > 0, `Missing ELF relocation coordinate 0x${key}`);
counts.set(key, remaining - 1);
}
const providerOnly = [...counts]
.filter(([, count]) => count > 0)
.map(([relative_address, count]) => ({
relative_address: `0x${relative_address}`,
count,
}));
const facts = {
observation_sha256: hash(JSON.stringify(observed)),
verification: {
status: source.status,
...(source.status === "unsupported"
? { reason: source.reason }
: { format: source.format }),
},
counts: {
source_files: observed.source_files.length,
mappings: observed.mappings.length,
relocations: observed.relocations.length,
entry_points: observed.entry_points.length,
entry_context: observed.entry_context.length,
metadata_recovery: observed.metadata_recovery?.length ?? null,
checks: source.checks?.length ?? null,
sections: source.sections?.length ?? null,
},
records: selectedRecords(observed.relocations),
};
if (values.mode === "summary") {
assert.equal(delivered.parent_operation, parent.operation);
assert.equal(delivered.parent_digest, parent.evidence_id.slice(3));
assert.equal(delivered.summary.verification.status, source.status);
assert.deepEqual(delivered.summary.counts, facts.counts);
assert.ok(!Object.hasOwn(delivered.summary.observation, "relocations"));
const selected = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: {
kind: "native-load-image",
collection: "relocations",
offset: 0,
limit: 4,
fields: ["address", "status"],
},
},
`${temperature}:selected-relocations`,
);
assert.deepEqual(
selected.normalized_result.items.rows,
facts.records.map((record, source_index) => ({ source_index, record })),
);
assert.equal(
selected.normalized_result.verification_status,
source.status,
);
assert.equal(
selected.normalized_result.coverage.total,
observed.relocations.length,
);
const tail = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: {
kind: "native-load-image",
collection: "relocations",
indices: [observed.relocations.length - 1],
fields: ["address", "original_bytes_hex", "memory_bytes_hex"],
},
},
`${temperature}:tail-relocation`,
);
const last = observed.relocations.at(-1);
assert.deepEqual(tail.normalized_result.items.rows, [
{
source_index: observed.relocations.length - 1,
record: {
address: last.address,
original_bytes_hex: last.original_bytes_hex,
memory_bytes_hex: last.memory_bytes_hex,
},
},
]);
const checks = await result(
"inspect_analysis_view",
{
source: { kind: "retained-evidence", evidence_id: id },
view: { kind: "native-load-image", collection: "checks", limit: 1 },
},
`${temperature}:verification-checks`,
);
assert.equal(source.status, "unsupported");
assert.equal(checks.normalized_result.items.rows, null);
}
assert.deepEqual(parent.parameters, {});
await writeFile(
join(values.output, `${temperature}-facts.json`),
JSON.stringify(facts),
);
if (values.control)
assert.deepEqual(
facts,
JSON.parse(
await readFile(
join(values.control, `${temperature}-facts.json`),
"utf8",
),
),
);
report.runs.push({
temperature,
parent_evidence_id: id,
source_provider: parent.provider,
source_analysis_profile_digest: parent.analysis_profile?.digest ?? null,
reply_refused: reply.isError === true,
complete_evidence_bytes: Buffer.byteLength(JSON.stringify(parent)),
facts_sha256: hash(JSON.stringify(facts)),
provider_only_relocation_coordinates: providerOnly,
elapsed_ms: performance.now() - runStart,
});
}
assert.equal(hash(await readFile(values.target)), targetSha256);
report.status = "passed";
} catch (cause) {
report.status = "failed";
report.failure = String(cause?.stack ?? cause);
process.exitCode = 1;
} finally {
if (opened) {
try {
await result("close_binary", {});
} catch (cause) {
report.status = "failed";
report.close_failure = String(cause);
process.exitCode = 1;
}
}
report.elapsed_before_finalization_ms = performance.now() - started;
if (
!(await finalizeRetainedQueryReceipt({
report,
client,
transport,
account: isolatedHome,
serverLog,
serverLogFinished,
output: values.output,
transcript,
}))
)
process.exitCode = 1;
}
+2 -2
View File
@@ -10,7 +10,7 @@ import {
toolContract,
} from "../dist/contracts/toolContracts.js";
import { advertisedInputJsonSchema } from "../dist/contracts/toolSchemaMetadata.js";
import { presentAdvertisedInputSchema } from "../dist/contracts/compactInputSchema.js";
import { presentAdvertisedInputProfile } from "../dist/contracts/inputSchemaPresentation.js";
import { compactPresentationForMcpInputSchemaProfile } from "../dist/config/mcpInputSchemaProfile.js";
import { createVerifierCommands } from "./lib/verify-commands.mjs";
import { withVerifierCleanup } from "./lib/verifier-cleanup.mjs";
@@ -186,7 +186,7 @@ await withVerifierCleanup(async () => {
);
assert.deepEqual(
tool.parameters,
presentAdvertisedInputSchema(
presentAdvertisedInputProfile(
advertisedInputJsonSchema(
toolContract(tool.name.slice(prefix.length)),
),
@@ -0,0 +1,320 @@
import { expect, it } from "vitest";
import { createEvidence } from "../../domain/evidence.js";
import {
analysisViewResultSchema,
inspectAnalysisViewInputSchema,
} from "../../domain/analysisView/analysisView.js";
import { projectNativeLoadImageView } from "../../domain/analysisView/nativeLoadImageView.js";
import type { NativeLoadImage } from "../../domain/native/nativeLoadImage.js";
import {
nativeDosLoadImageFixture,
nativeLoadImageObservationFixture,
} from "../../../tests/fixtures/nativeLoadImage.js";
import { inspectAnalysisView } from "./AnalysisViewService.js";
const source = (): Extract<NativeLoadImage, { status: "unsupported" }> => ({
status: "unsupported",
reason:
"ELF load observations have no independent verification in this fixture.",
observations: {
...nativeLoadImageObservationFixture(),
executable_format: "Executable and Linking Format (ELF)",
relocations: [
{
address: "ram:0x0010",
segment: null,
offset: null,
status: "APPLIED",
type: 8,
values: [16],
original_bytes_hex: "0000",
memory_bytes_hex: "1000",
},
{
address: "ram:0x0010",
segment: null,
offset: null,
status: "UNSUPPORTED",
type: 9,
values: [],
original_bytes_hex: null,
memory_bytes_hex: null,
},
],
},
limitations: [
"Load observations do not establish runtime relocation behavior.",
],
});
const evidence = (report: NativeLoadImage) =>
createEvidence(
{ path: "/selected/fixture", sha256: "a".repeat(64), format: "elf" },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "inspect_native_load_image",
parameters: {},
result: report,
limitations: ["Provider observation remains historical."],
},
{ immutable: true },
);
const view = async (report: NativeLoadImage, request: unknown) => {
const result = await inspectAnalysisView({
source: { kind: "inline", evidence: evidence(report) },
view: request,
});
if (!result.ok) throw result.error;
return analysisViewResultSchema.parse(result.value.normalized_result);
};
it("summarizes measured ELF facts without upgrading unsupported verification", async () => {
const report = source();
const parent = evidence(report);
const summary = await view(report, { kind: "summary" });
expect(summary).toMatchObject({
parent_evidence_id: parent.evidence_id,
parent_operation: "inspect_native_load_image",
source_provider: parent.provider,
source_analysis_profile_digest: null,
source_parameters: {},
artifact: { path: "/selected/fixture", sha256: "a".repeat(64) },
summary: {
verification: { status: "unsupported", reason: report.reason },
observation: {
executable_format: "Executable and Linking Format (ELF)",
},
counts: {
relocations: 2,
checks: null,
sections: null,
metadata_recovery: null,
},
},
limitations: expect.arrayContaining([
...report.limitations,
"Provider observation remains historical.",
]),
unknowns: expect.arrayContaining([report.reason]),
});
expect(JSON.stringify(summary)).not.toContain("memory_bytes_hex");
});
it("preserves same-address alternatives, padding, nulls and byte facts over pages", async () => {
const report = source();
const records = [];
for (let offset = 0; offset < 2; offset++) {
const page = await view(report, {
kind: "native-load-image",
collection: "relocations",
fields: [
"address",
"segment",
"offset",
"status",
"type",
"values",
"original_bytes_hex",
"memory_bytes_hex",
],
offset,
limit: 1,
});
if (page.kind !== "native-load-image") throw new Error("Expected page");
expect(page.verification_status).toBe("unsupported");
expect(
analysisViewResultSchema.safeParse({
...page,
items: {
...page.items,
rows: null,
},
}).success,
).toBe(false);
expect(page.items.rows?.[0]?.source_index).toBe(offset);
expect(page.coverage).toMatchObject({
total: 2,
next_offset: offset === 0 ? 1 : null,
exhausted: offset === 1,
});
records.push(...(page.items.rows ?? []).map((row) => row.record));
}
expect(records).toEqual(report.observations.relocations);
});
it("filters indices before paging and projects only selected fields", async () => {
const page = await view(source(), {
kind: "native-load-image",
collection: "relocations",
indices: [1, 400, 0, 1],
offset: 1,
fields: ["address", "status", "status"],
limit: 1,
});
expect(page).toMatchObject({
items: {
collection: "relocations",
rows: [
{
source_index: 1,
record: { address: "ram:0x0010", status: "UNSUPPORTED" },
},
],
},
coverage: { examined: 1, total: 2, next_offset: null, exhausted: true },
});
if (page.kind !== "native-load-image") throw new Error("Expected page");
expect(page.items.rows?.[0]?.record).toEqual({
address: "ram:0x0010",
status: "UNSUPPORTED",
});
});
it.each(["checks", "sections"])(
"keeps unrecorded or inapplicable %s null rather than inventing empty observations",
async (collection) => {
expect(
await view(source(), { kind: "native-load-image", collection }),
).toMatchObject({
items: { collection, rows: null },
unknowns: expect.arrayContaining([
"Null collection counts or rows mean not recorded or not applicable, not observed absence.",
]),
});
},
);
it("distinguishes an explicitly recorded empty metadata collection and exhausted selection", async () => {
const report = source();
report.observations.metadata_recovery = [];
expect(await view(report, { kind: "summary" })).toMatchObject({
summary: { counts: { metadata_recovery: 0 } },
});
for (const request of [
{ collection: "entry_context" },
{ collection: "relocations", indices: [] },
{ collection: "relocations", offset: 400 },
])
expect(
await view(report, { kind: "native-load-image", ...request }),
).toMatchObject({
items: { rows: [] },
coverage: { status: "empty", exhausted: true, next_offset: null },
});
});
it("preserves recorded DOS verification details and ordered mismatch checks", async () => {
const report = nativeDosLoadImageFixture();
const summary = await view(report, { kind: "summary" });
expect(summary).toMatchObject({
summary: {
verification: {
status: "verified",
format: "dos-mz",
target_sha256: report.target_sha256,
},
counts: { checks: report.checks.length },
},
});
const mismatch = {
...report,
status: "mismatch" as const,
checks: [
{
...report.checks[0]!,
matched: false,
expected: { original: 1 },
observed: null,
},
],
};
expect(
await view(mismatch, {
kind: "native-load-image",
collection: "checks",
fields: [
"name",
"matched",
"expected",
"observed",
"file_offset",
"address",
],
}),
).toMatchObject({
items: { rows: [{ source_index: 0, record: mismatch.checks[0] }] },
});
expect(await view(mismatch, { kind: "summary" })).toMatchObject({
summary: { verification: { status: "mismatch" } },
});
});
it("keeps byte and potentially large comparison payloads opt-in while retaining full source facts", async () => {
const relocations = await view(source(), {
kind: "native-load-image",
collection: "relocations",
});
if (relocations.kind !== "native-load-image")
throw new Error("Expected page");
expect(relocations.items.rows?.[0]?.record).toEqual({
address: "ram:0x0010",
segment: null,
offset: null,
status: "APPLIED",
type: 8,
values: [16],
});
const report = nativeDosLoadImageFixture();
const body = { complete_comparison: "retained proof body".repeat(128) };
report.checks[0]!.expected = body;
const checks = await view(report, {
kind: "native-load-image",
collection: "checks",
});
if (checks.kind !== "native-load-image") throw new Error("Expected page");
expect(checks.items.rows?.[0]?.record).toEqual({
name: "original_file_sha256",
matched: true,
file_offset: null,
address: null,
});
expect(
await view(report, {
kind: "native-load-image",
collection: "checks",
fields: ["expected"],
}),
).toMatchObject({
items: { rows: [{ source_index: 0, record: { expected: body } }] },
});
expect(report.checks[0]!.expected).toEqual(body);
});
it("rejects collection-inapplicable fields and malformed sibling facts even for an empty selection", async () => {
expect(
inspectAnalysisViewInputSchema.safeParse({
source: { kind: "inline", evidence: evidence(source()) },
view: {
kind: "native-load-image",
collection: "entry_points",
fields: ["original_bytes_hex"],
},
}).success,
).toBe(false);
const report = source();
const parent = {
evidenceId: "ev_" + "b".repeat(64),
operation: "inspect_native_load_image",
normalizedResult: report,
limitations: [],
};
const request = {
kind: "native-load-image" as const,
collection: "relocations" as const,
indices: [],
offset: 0,
limit: 1,
};
expect(projectNativeLoadImageView(parent, request).ok).toBe(true);
Reflect.set(report.observations.relocations[1]!, "status", "");
expect(projectNativeLoadImageView(parent, request).ok).toBe(false);
});
+3 -5
View File
@@ -12,10 +12,8 @@ import {
advertisedInputJsonSchema,
toolOutputSchemaWithMetadata,
} from "./contracts/toolSchemaMetadata.js";
import {
presentAdvertisedInputSchema,
type CompactInputSchemaPresentation,
} from "./contracts/compactInputSchema.js";
import type { CompactInputSchemaPresentation } from "./contracts/compactInputSchema.js";
import { presentAdvertisedInputProfile } from "./contracts/inputSchemaPresentation.js";
import type { McpOutputSchemaProfile } from "./config/mcpOutputSchemaProfile.js";
import {
COMPACT_INPUT_SCHEMA_BUDGET_BYTES,
@@ -62,7 +60,7 @@ const presentToolCatalog = (
? catalog
: catalog.map((tool) => ({
...tool,
input_schema: presentAdvertisedInputSchema(
input_schema: presentAdvertisedInputProfile(
tool.input_schema,
presentation,
),
+12 -2
View File
@@ -1,7 +1,11 @@
import { readArtifactInspectionSources } from "../domain/artifactInspection.js";
import type { AnalysisViewRequest } from "../domain/analysisView/analysisView.js";
import { z } from "incur";
import { parseEvidence, type Evidence } from "../domain/evidence.js";
import {
parseEvidence,
withEvidenceCallerInputPath,
type Evidence,
} from "../domain/evidence.js";
import type { JsonValue } from "../domain/jsonValue.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import type { AnalysisError } from "../domain/analysisErrorBase.js";
@@ -180,6 +184,12 @@ export const publishCliAnalysisEvidence = async (
return projectAnalysisError(
delivered.error.retainPartialObservation(complete),
);
// This projection answers the same direct invocation as its parent. Later
// inspect_analysis_view calls own their own input and do not inherit it.
const selected =
detail === "summary" && complete.caller_input_path !== undefined
? withEvidenceCallerInputPath(delivered.value, complete.caller_input_path)
: delivered.value;
const inspectionSources =
complete.operation === "inspect_artifact"
? readArtifactInspectionSources(complete.normalized_result)
@@ -199,7 +209,7 @@ export const publishCliAnalysisEvidence = async (
).values(),
]),
complete,
delivered: delivered.value,
delivered: selected,
outputPath: absoluteCallerPath(evidenceOutput),
operation,
signal,
+28 -8
View File
@@ -16,6 +16,13 @@ import {
snapshotSelectionOption,
} from "./options.js";
import type { CliInstance } from "./types.js";
import { withCommandCancellation } from "./commandCancellation.js";
import {
analysisEvidenceDeliveryOptions,
cliEvidenceDeliveryInputError,
publishCliAnalysisEvidence,
} from "./analysisEvidenceDelivery.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
/** Register native load-image, memory, and annotation CLI commands. */
export const registerCoreNativeCommands = (
@@ -66,19 +73,32 @@ export const registerCoreNativeCommands = (
});
cli.command(CLI_COMMANDS.inspectNativeLoadImage, {
description:
"Verify loaded native bytes, source mappings, relocations and entry",
"Inspect native load observations and supported snapshot verification",
args: z.object({ path: z.string().describe("Local executable path") }),
options: z.object({
...analysisEvidenceDeliveryOptions,
...nativeOpenOptionFields,
}),
run: ({ args, options }) =>
logCliCommand(logger, CLI_COMMANDS.inspectNativeLoadImage, () =>
runDirectAnalysis(
args.path,
"inspect_native_load_image",
{},
directAnalysisOptionsFromCli(logger, options),
),
withCommandCancellation((signal) =>
logCliCommand(logger, CLI_COMMANDS.inspectNativeLoadImage, async () => {
const delivery = {
providerId: "auto",
operation: "inspect_native_load_image",
signal,
detail: options.detail,
evidenceOutput: options.evidence_output,
};
const invalid = cliEvidenceDeliveryInputError(delivery);
if (invalid !== undefined) return projectAnalysisError(invalid);
const complete = await runDirectAnalysis(
args.path,
"inspect_native_load_image",
{},
{ ...directAnalysisOptionsFromCli(logger, options), signal },
);
return publishCliAnalysisEvidence(complete, delivery);
}),
),
});
cli.command(CLI_COMMANDS.readBytes, {
+1 -1
View File
@@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [
...toolContractMetadata("inspect_analysis_view"),
kind: "application",
description:
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where viewLine truncated
'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where viewLine truncated
inputSchema: inspectAnalysisViewInputSchema,
outputSchema: evidenceResultOf(analysisViewResultSchema),
examples: [
@@ -0,0 +1,264 @@
import { Ajv } from "ajv";
import { Ajv2020 } from "ajv/dist/2020.js";
import { expect, it } from "vitest";
import { advertisedInputExamples } from "./advertisedInputExamples.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
fitAdvertisedInputSchema,
presentInputJsonSchema,
summarizedInputSchemaCandidate,
summarizedSchema,
} from "./inputSchemaPresentation.js";
import {
schemaArrays,
schemaChildren,
schemaLiteralValues,
schemaMaps,
} from "./jsonSchemaKeywords.js";
import { TOOL_CONTRACTS } from "./toolContracts.js";
type Schema = Record<string, unknown>;
const bytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
const isObject = (value: unknown): value is Schema =>
typeof value === "object" && value !== null && !Array.isArray(value);
/** The self-contained projection before fitting, with its advertised examples. */
const completeProjection = (
contract: (typeof TOOL_CONTRACTS)[number],
): Schema => {
const inline = contract.inputSchema["~standard"].jsonSchema.input({
target: "draft-2020-12",
libraryOptions: { reused: "inline" },
}) as Schema;
const examples = advertisedInputExamples(contract);
return {
...presentInputJsonSchema(inline, () => undefined),
...(examples.length === 0 ? {} : { examples }),
};
};
/** Every subschema in a schema position, ignoring literal data. */
const subschemas = function* (schema: unknown): Generator<Schema> {
if (!isObject(schema)) return;
yield schema;
for (const [key, child] of Object.entries(schema)) {
if (schemaLiteralValues.has(key)) continue;
if (schemaMaps.has(key) && isObject(child))
for (const nested of Object.values(child)) yield* subschemas(nested);
else if (schemaArrays.has(key) && Array.isArray(child))
for (const nested of child) yield* subschemas(nested);
else if (schemaChildren.has(key))
for (const nested of Array.isArray(child) ? child : [child])
yield* subschemas(nested);
}
};
/** Every JSON value nested in a value, including the value itself. */
const values = function* (value: unknown): Generator<unknown> {
yield value;
if (Array.isArray(value)) for (const item of value) yield* values(item);
else if (isObject(value))
for (const item of Object.values(value)) yield* values(item);
};
const GENERIC_PROBES: readonly unknown[] = [
null,
true,
false,
0,
1,
-1,
1.5,
"",
"a",
[],
[null],
[1],
["a"],
["a", 1],
[[]],
[{}],
{},
{ kind: "a" },
];
/** Assert a summary accepts every probe the complete schema accepts. */
const expectSummaryWidens = (
schemas: Iterable<Schema>,
probes: readonly unknown[],
dialect: "draft-2020-12" | "draft-07" = "draft-2020-12",
): number => {
const options = { strict: false, validateFormats: false };
const ajv = dialect === "draft-07" ? new Ajv(options) : new Ajv2020(options);
const seen = new Set<string>();
const failures: string[] = [];
for (const schema of schemas) {
const key = JSON.stringify(schema);
if (seen.has(key)) continue;
seen.add(key);
const summary = summarizedSchema(schema);
const complete = ajv.compile(schema);
const summarized = ajv.compile(summary);
for (const probe of probes)
if (complete(probe) && !summarized(probe))
failures.push(
`${key.slice(0, 200)} -> ${JSON.stringify(summary)} rejects ${JSON.stringify(probe).slice(0, 200)}`,
);
ajv.removeSchema();
}
expect(failures).toEqual([]);
return seen.size;
};
const SYNTHETIC_SUMMARIES: readonly Schema[] = [
// 2020-12 tuple positions with a differently typed rest.
{
type: "array",
prefixItems: [{ type: "string" }],
items: { type: "number" },
},
{ type: "array", prefixItems: [{ type: "string" }] },
{ type: "array", prefixItems: [{ type: "string" }], items: false },
{ type: "array", items: { type: "string" }, contains: { const: "a" } },
{
anyOf: [
{ type: "array", prefixItems: [{ type: "boolean" }] },
{ type: "null" },
],
},
{ type: ["array", "null"], items: { type: "string" }, minItems: 1 },
{ type: "object", properties: { a: { type: "string" } }, required: ["a"] },
{ not: { type: "string" } },
{ allOf: [{ type: "string" }, { minLength: 1 }] },
{ oneOf: [{ const: 1 }, { enum: ["a", null] }] },
{
if: { type: "string" },
then: { minLength: 1 },
else: { type: "number" },
},
];
const DRAFT_07_TUPLES: readonly Schema[] = [
{
type: "array",
items: [{ type: "string" }],
additionalItems: { type: "number" },
},
{ type: "array", items: [{ type: "string" }] },
{ type: "array", items: [{ type: "string" }], additionalItems: false },
];
it("summarizes synthetic schemas without rejecting values they accept", () => {
expect(
summarizedSchema({
type: "array",
prefixItems: [{ type: "string" }],
items: { type: "number" },
}),
).toEqual({ type: "array", items: { type: ["string", "number"] } });
expectSummaryWidens(SYNTHETIC_SUMMARIES, [
...GENERIC_PROBES,
[true],
["a", "b", 2],
[1, "a"],
[false, null],
]);
expectSummaryWidens(
DRAFT_07_TUPLES,
[...GENERIC_PROBES, ["a", 2], ["a", "b"]],
"draft-07",
);
});
it("summarizes every catalog input subschema without rejecting values it accepts", () => {
const projections = TOOL_CONTRACTS.map(completeProjection);
// Canonical example values reach deep nested subschemas with valid data.
const probes = [
...GENERIC_PROBES,
...new Map(
projections
.flatMap((schema) =>
Array.isArray(schema.examples) ? schema.examples : [],
)
.flatMap((example) => [...values(example)])
.map((value) => [JSON.stringify(value), value] as const),
).values(),
];
const checked = expectSummaryWidens(
projections.flatMap((schema) =>
[...subschemas(schema)].filter((node) => node !== schema),
),
probes,
);
expect(checked).toBeGreaterThan(1000);
}, 120_000);
/** Assert no candidate keeping more levels than the fitted one also fits. */
const expectDeepestFittingCandidate = (schema: Schema): void => {
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const described = { ...schema, description: fitted.description };
const candidates = Array.from({ length: 40 }, (_, index) =>
summarizedInputSchemaCandidate(described, index + 1),
);
const deepest = candidates.findLastIndex(
(candidate) => bytes(candidate) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(deepest).toBeGreaterThanOrEqual(0);
expect(fitted).toEqual(candidates[deepest]);
};
it("keeps a deeper fitting candidate when a shallower summary is longer", () => {
// A nullable array summary needs anyOf, so it is longer than the schema.
const nullable = { type: ["array", "null"], items: { type: "string" } };
const schema = {
type: "object",
properties: {
...Object.fromEntries(
Array.from({ length: 1_000 }, (_, index) => [
`list_${index}`,
nullable,
]),
),
// Retaining its third level would overflow; its second level fits.
large: {
type: "object",
properties: {
inner: {
type: "object",
properties: {
text: { type: "string", description: "x".repeat(30_000) },
},
},
},
},
},
};
// Summarizing every list at the first level would overflow by itself.
expect(1_000 * bytes(summarizedSchema(nullable))).toBeGreaterThan(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const properties = fitted.properties as Record<string, any>;
expect(properties.list_0).toEqual(nullable);
expect(properties.large.properties.inner).toEqual({ type: "object" });
expectDeepestFittingCandidate(schema);
});
it("fits oversized catalog input schemas at their deepest fitting level", () => {
const oversized = TOOL_CONTRACTS.map(completeProjection).filter(
(schema) => bytes(schema) > ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(oversized.length).toBeGreaterThan(0);
for (const schema of oversized) expectDeepestFittingCandidate(schema);
}, 120_000);
@@ -0,0 +1,371 @@
import { Ajv2020 } from "ajv/dist/2020.js";
import { describe, expect, it } from "vitest";
import { presentAdvertisedInputSchema } from "./compactInputSchema.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
fitAdvertisedInputSchema,
presentAdvertisedInputProfile,
presentInputJsonSchema,
} from "./inputSchemaPresentation.js";
const bytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
const ajv = () => new Ajv2020({ strict: false, validateFormats: false });
/** A strict object `levels` deep with `width` distinct padded string leaves. */
const record = (
levels: number,
width: number,
pad = 80,
path = "f",
): Record<string, unknown> => ({
type: "object",
properties: Object.fromEntries(
Array.from({ length: width }, (_, index) => [
`field_${index}`,
levels <= 1
? {
type: "string",
minLength: 1,
description: `Leaf ${path}.${index} ${"x".repeat(pad)}`,
}
: {
...record(levels - 1, width, pad, `${path}.${index}`),
description: `Level ${index}`,
},
]),
),
required: ["field_0"],
additionalProperties: false,
});
const instance = (levels: number, width: number): Record<string, any> =>
Object.fromEntries(
Array.from({ length: width }, (_, index) => [
`field_${index}`,
levels <= 1 ? "value" : instance(levels - 1, width),
]),
);
const expectWidened = (
original: Record<string, unknown>,
fitted: Record<string, unknown>,
inputs: readonly unknown[],
) => {
const validateOriginal = ajv().compile(original);
const validateFitted = ajv().compile(fitted);
for (const input of inputs)
if (validateOriginal(input))
expect(validateFitted(input), JSON.stringify(validateFitted.errors)).toBe(
true,
);
};
it("returns schemas that already fit unchanged", () => {
const schema = {
type: "object",
properties: { capture: record(3, 4) },
examples: [{ capture: instance(3, 4) }],
};
expect(bytes(schema)).toBeLessThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
expect(fitAdvertisedInputSchema(schema)).toBe(schema);
});
it("points repeated sibling properties at the first instead of repeating them", () => {
const capture = record(3, 7, 40);
const schema = {
type: "object",
description: "Compare two captures.",
properties: {
before: { ...capture, description: "Earlier capture." },
after: { ...capture, description: "Later capture." },
unrelated: { type: "string" },
},
required: ["before", "after"],
additionalProperties: false,
examples: [{ before: instance(3, 7), after: instance(3, 7) }],
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
expect(bytes(capture)).toBeLessThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(fitted).toMatchObject({
description: expect.stringMatching(
/^Compare two captures\. .*summarized form.*complete canonical inputs\.$/u,
),
properties: {
before: schema.properties.before,
after: {
type: "object",
description: "Later capture. Same schema as `before`.",
},
unrelated: { type: "string" },
},
required: ["before", "after"],
additionalProperties: false,
examples: schema.examples,
});
expect(Object.keys(fitted.properties as object)).toEqual([
"before",
"after",
"unrelated",
]);
expect(ajv().validateSchema(fitted)).toBe(true);
expectWidened(schema, fitted, [
...schema.examples,
{ before: instance(3, 7), after: { field_0: { field_0: {} } } },
]);
});
it("keeps the deepest instance level that fits and summarizes below it", () => {
const schema = {
type: "object",
properties: { capture: record(4, 6) },
additionalProperties: false,
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const capture = (fitted.properties as Record<string, any>).capture;
// Third-level objects keep their properties; fourth-level ones are summarized.
const third = capture.properties.field_1.properties.field_2;
expect(third).toMatchObject({
required: ["field_0"],
additionalProperties: false,
});
expect(third.properties.field_3).toEqual({
type: "object",
description: "Level 3",
});
expectWidened(schema, fitted, [
{ capture: instance(4, 6) },
{ capture: { field_0: { field_0: { field_0: { field_0: "" } } } } },
]);
});
it("counts combinator and conditional branches at their instance level", () => {
const branch = (kind: string) => ({
type: "object",
properties: {
kind: { type: "string", enum: [kind] },
body: record(3, 9),
},
required: ["kind", "body"],
additionalProperties: false,
});
const schema = {
type: "object",
properties: {
value: { anyOf: [branch("a"), branch("b"), { type: "null" }] },
},
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
const value = (fitted.properties as Record<string, any>).value;
// The union keeps every branch and its own properties at level two.
expect(value.anyOf).toHaveLength(3);
expect(value.anyOf[0].properties.kind).toEqual({
type: "string",
enum: ["a"],
});
expect(value.anyOf[2]).toEqual({ type: "null" });
});
it("relaxes nonmonotone positions instead of narrowing them", () => {
const deep = record(3, 7);
const schema = {
type: "object",
properties: {
exactly_one: {
oneOf: [
{ ...deep, required: ["field_0"] },
{ ...deep, required: ["field_1"] },
],
},
excluded: { type: "object", not: deep },
conditional: {
type: "object",
if: deep,
then: { required: ["field_0"] },
else: { required: ["field_1"] },
},
bounded: { type: "array", contains: deep, maxContains: 1 },
},
};
expect(bytes(schema)).toBeGreaterThan(ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES);
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
const properties = fitted.properties as Record<string, any>;
expect(properties.exactly_one.oneOf).toBeUndefined();
expect(properties.exactly_one.anyOf).toHaveLength(2);
expect(properties.excluded).not.toHaveProperty("not");
expect(properties.conditional).not.toHaveProperty("if");
expect(properties.conditional).not.toHaveProperty("then");
expect(properties.bounded).not.toHaveProperty("maxContains");
const valid = instance(3, 7);
expectWidened(schema, fitted, [
{ exactly_one: { field_0: valid.field_0 } },
{ exactly_one: { field_1: valid.field_1 } },
{ excluded: { other: true } },
{ conditional: { field_1: valid.field_1 } },
{ conditional: valid },
{ bounded: [valid, { field_0: 1 }] },
]);
});
it("advertises items only beside the single array type", () => {
const rows = {
type: "array",
items: { type: "object", properties: { id: { type: "string" } } },
minItems: 1,
description: "Rows.",
};
const schema = {
type: "object",
properties: {
// Its large leaf one level deeper limits the retained depth to two.
large: {
type: "object",
properties: {
inner: {
type: "object",
properties: {
text: { type: "string", description: "x".repeat(70_000) },
},
},
},
},
arrays: {
type: "object",
properties: {
rows,
optional_rows: { anyOf: [rows, { type: "null" }] },
cube: { type: "array", items: { type: "array", items: rows } },
},
},
},
};
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect((fitted.properties as Record<string, any>).arrays).toEqual({
type: "object",
properties: {
rows: {
type: "array",
items: { type: "object" },
minItems: 1,
description: "Rows.",
},
optional_rows: {
anyOf: [{ type: "array", items: { type: "object" } }, { type: "null" }],
},
cube: { type: "array", items: {} },
},
});
expectWidened(schema, fitted, [
{
arrays: {
rows: [{ id: "a" }],
optional_rows: null,
cube: [[[{ id: "c" }]]],
},
},
{ arrays: { optional_rows: [{ id: "b" }] } },
]);
});
it("falls back to the compact projection for schemas with local references", () => {
const schema = {
type: "object",
$defs: { capture: record(3, 12) },
properties: {
before: { $ref: "#/$defs/capture" },
after: { $ref: "#/$defs/capture" },
},
};
const fitted = fitAdvertisedInputSchema(schema);
expect(bytes(fitted)).toBeLessThanOrEqual(
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
expect(JSON.stringify(fitted)).not.toContain("$ref");
expect(ajv().validateSchema(fitted)).toBe(true);
});
it("presents the compact profile from the complete projection", () => {
const capture = record(3, 7);
const complete = {
type: "object",
properties: {
before: { ...capture, description: "Earlier capture." },
after: { ...capture, description: "Later capture." },
},
};
const fitted = fitAdvertisedInputSchema(complete);
expect(fitted).not.toEqual(complete);
const budget = { budgetBytes: 13 * 1024 };
expect(presentAdvertisedInputProfile(fitted, budget)).toEqual(
presentAdvertisedInputSchema(complete, budget),
);
expect(presentAdvertisedInputProfile(fitted, undefined)).toBe(fitted);
});
describe("advertised numeric literals", () => {
it("relaxes numeric const and enum members without narrowing", () => {
const schema = {
type: "object",
properties: {
version: { type: "integer", const: 2 },
ratio: { const: 0.5 },
level: { enum: [1, 2, 3] },
mixed: { enum: ["auto", 7] },
not_two: { type: "integer", not: { const: 2 } },
},
};
const presented = presentInputJsonSchema(schema, () => undefined);
expect(presented.properties).toEqual({
version: { type: "integer" },
ratio: { type: "number" },
level: { type: "integer" },
mixed: { type: ["string", "integer"] },
not_two: { type: "integer" },
});
expectWidened(schema, presented, [
{ version: 2, ratio: 0.5, level: 3, mixed: "auto", not_two: 3 },
{ mixed: 7 },
]);
});
it("drops unevaluated keywords when relaxing an applicator they depend on", () => {
const schema = {
type: "object",
properties: {
options: {
type: "object",
properties: { mode: { type: "integer" } },
if: { properties: { mode: { const: 2 } }, required: ["mode"] },
then: { properties: { extra: { type: "string" } } },
unevaluatedProperties: false,
},
},
};
const presented = presentInputJsonSchema(schema, () => undefined);
expect(presented.properties).toEqual({
options: { type: "object", properties: { mode: { type: "integer" } } },
});
expectWidened(schema, presented, [
{ options: { mode: 2, extra: "kept" } },
{ options: { mode: 1 } },
]);
});
});
+423 -46
View File
@@ -9,6 +9,11 @@ import {
schemaChildren,
schemaMaps,
} from "./jsonSchemaKeywords.js";
import {
compactAdvertisedInputSchema,
presentAdvertisedInputSchema,
type CompactInputSchemaPresentation,
} from "./compactInputSchema.js";
const isObject = (value: unknown): value is Readonly<Record<string, unknown>> =>
typeof value === "object" && value !== null && !Array.isArray(value);
@@ -155,6 +160,223 @@ type SchemaNormalization = {
readonly changed: boolean;
};
const serializedBytes = (value: unknown): number =>
Buffer.byteLength(JSON.stringify(value), "utf8");
/** Instance nesting levels below the current node that keep their structure. */
type SchemaSummary = { readonly levels: number };
// Keywords whose subschemas describe a nested instance. Combinators,
// conditionals and dependent schemas describe the same instance as their parent.
const instanceChildKeys = new Set([
"properties",
"patternProperties",
"additionalProperties",
"unevaluatedProperties",
"propertyNames",
"prefixItems",
"items",
"additionalItems",
"contains",
"unevaluatedItems",
]);
const hasNestedSchema = (schema: Readonly<Record<string, unknown>>): boolean =>
Object.entries(schema).some(([key, child]) =>
schemaMaps.has(key)
? isObject(child) && Object.values(child).some(isObject)
: schemaArrays.has(key)
? Array.isArray(child) && child.length > 0
: schemaChildren.has(key) && isObject(child),
);
/** Count nested instance levels a schema describes, ignoring literal data. */
const schemaLevels = (schema: unknown): number => {
if (!isObject(schema)) return 0;
let deepest = 0;
for (const [key, child] of Object.entries(schema)) {
const children = schemaMaps.has(key)
? isObject(child)
? Object.values(child)
: []
: schemaArrays.has(key)
? Array.isArray(child)
? child
: []
: schemaChildren.has(key)
? [child]
: [];
const step = instanceChildKeys.has(key) ? 1 : 0;
for (const nested of children)
if (isObject(nested))
deepest = Math.max(deepest, step + schemaLevels(nested));
}
return deepest;
};
const literalType = (value: unknown): string => {
if (value === null) return "null";
if (Array.isArray(value)) return "array";
if (typeof value === "number")
return Number.isInteger(value) ? "integer" : "number";
return typeof value;
};
const typeKeyword = (types: readonly string[]): string | string[] =>
types.length === 1 ? types[0]! : [...types];
/** Every JSON type a schema can accept, or undefined when that is not explicit. */
const schemaTypes = (schema: unknown): readonly string[] | undefined => {
if (!isObject(schema)) return undefined;
if (typeof schema.type === "string") return [schema.type];
if (
Array.isArray(schema.type) &&
schema.type.length > 0 &&
schema.type.every((type) => typeof type === "string")
)
return schema.type;
const literals = Object.hasOwn(schema, "const")
? [schema.const]
: Array.isArray(schema.enum) && schema.enum.length > 0
? schema.enum
: undefined;
if (literals !== undefined) return [...new Set(literals.map(literalType))];
const branches = Array.isArray(schema.anyOf)
? schema.anyOf
: Array.isArray(schema.oneOf)
? schema.oneOf
: [];
const types = branches.map(schemaTypes);
return types.length > 0 && types.every((type) => type !== undefined)
? [...new Set(types.flat())]
: undefined;
};
/** Item types of an explicitly typed array, including tuple positions. */
const typedArrayItemTypes = (
schema: Readonly<Record<string, unknown>>,
): readonly string[] | undefined => {
// Tuple positions come from `prefixItems` (2020-12) or an `items` array
// (draft-07); later elements from `items` or `additionalItems`.
const positions = [
...(Array.isArray(schema.prefixItems) ? schema.prefixItems : []),
...(Array.isArray(schema.items) ? schema.items : []),
];
const rest = Array.isArray(schema.items)
? schema.additionalItems
: schema.items;
// An absent or `true` rest schema admits any later element.
if (rest !== false && !isObject(rest)) return undefined;
const types = [...positions, ...(rest === false ? [] : [rest])].map(
schemaTypes,
);
return types.every((type) => type !== undefined)
? [...new Set(types.flat())]
: undefined;
};
/** Every type an array accepted by the schema can contain, when explicit. */
const arrayItemTypes = (schema: unknown): readonly string[] | undefined => {
if (!isObject(schema)) return undefined;
const types = schemaTypes(schema);
if (types !== undefined && !types.includes("array")) return [];
if (schema.type !== undefined) return typedArrayItemTypes(schema);
const branches = Array.isArray(schema.anyOf)
? schema.anyOf
: Array.isArray(schema.oneOf)
? schema.oneOf
: [];
const itemTypes = branches.map(arrayItemTypes);
return itemTypes.length > 0 && itemTypes.every((type) => type !== undefined)
? [...new Set(itemTypes.flat())]
: undefined;
};
// Size constraints on the summarized instance itself. Required names stay
// with the properties they name: Gemini rejects required undeclared properties.
const summarizedConstraintKeys = [
"minProperties",
"maxProperties",
"minItems",
"maxItems",
] as const;
/**
* Replace a nested schema with its accepted types, own size constraints and
* guidance. The summary accepts every value the complete
* schema accepts, so it only widens the advertisement; `items` appears only
* beside the single `array` type.
*/
export const summarizedSchema = (
schema: Readonly<Record<string, unknown>>,
): Record<string, unknown> => {
const retained = {
...Object.fromEntries(
summarizedConstraintKeys.flatMap((key) =>
Object.hasOwn(schema, key) ? [[key, schema[key]]] : [],
),
),
...(typeof schema.description === "string"
? { description: schema.description }
: {}),
};
const types = schemaTypes(schema);
if (types === undefined) return retained;
const others = types.filter((type) => type !== "array");
if (others.length === types.length)
return { type: typeKeyword(types), ...retained };
const itemTypes = arrayItemTypes(schema);
const array = {
type: "array",
items:
itemTypes === undefined ||
itemTypes.length === 0 ||
itemTypes.includes("array")
? {}
: { type: typeKeyword(itemTypes) },
};
return others.length === 0
? { ...array, ...retained }
: { anyOf: [array, { type: typeKeyword(others) }], ...retained };
};
const withoutDescription = ({
description: _description,
...schema
}: Readonly<Record<string, unknown>>): Readonly<Record<string, unknown>> =>
schema;
/**
* Summarize a property whose nested schema repeats an earlier sibling's, such
* as `after` beside `before`, as a pointer to that sibling. Records the first
* occurrence of each nested schema in `complete`.
*/
const sharedSiblingSchema = (
name: string,
child: unknown,
complete: [string, Readonly<Record<string, unknown>>][],
): Record<string, unknown> | undefined => {
if (!isObject(child) || !hasNestedSchema(child)) return undefined;
const shape = withoutDescription(child);
const sibling = complete.find(([, known]) =>
isDeepStrictEqual(withoutDescription(known), shape),
)?.[0];
if (sibling === undefined) {
complete.push([name, child]);
return undefined;
}
const shared = {
...summarizedSchema(child),
description: [
typeof child.description === "string" ? child.description : undefined,
`Same schema as \`${sibling}\`.`,
]
.filter((text) => text !== undefined)
.join(" "),
};
return serializedBytes(shared) < serializedBytes(child) ? shared : undefined;
};
const isBooleanLiteralSchema = (schema: unknown): boolean =>
isObject(schema) &&
(typeof schema.const === "boolean" ||
@@ -182,36 +404,122 @@ const preserveSharedDefault = (
schema.default = defaults[0];
};
const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
/**
* Relax literal constraints that Gemini declarations reject, in place, without
* rejecting a value the literal accepted. Returns whether anything changed.
*/
const relaxUnsupportedLiterals = (node: Record<string, unknown>): boolean => {
let relaxed = false;
if (typeof node.const === "boolean") {
delete node.const;
if (node.type === undefined) node.type = "boolean";
relaxed = true;
}
if (node.const === "") {
delete node.const;
if (node.type === undefined) node.type = "string";
relaxed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.length > 0 &&
node.enum.every((item) => typeof item === "boolean")
) {
delete node.enum;
if (node.type === undefined) node.type = "boolean";
relaxed = true;
}
if (Array.isArray(node.enum) && node.enum.includes("")) {
// Removing only the empty member would reject a canonical-valid input.
// Gemini transports turn const into enum and refuse empty enum members.
const values = node.enum;
delete node.enum;
if (
node.type === undefined &&
values.every((item) => typeof item === "string")
)
node.type = "string";
relaxed = true;
}
if (typeof node.const === "number") {
// Gemini accepts only string enum members, and transports turn const into
// enum. The declared or literal numeric type still guides callers.
const literal = node.const;
delete node.const;
if (node.type === undefined) node.type = literalType(literal);
relaxed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.some((item) => typeof item === "number")
) {
const values = node.enum;
delete node.enum;
if (node.type === undefined)
node.type = typeKeyword([...new Set(values.map(literalType))]);
relaxed = true;
}
return relaxed;
};
const normalizeAdvertisedSchema = (
value: unknown,
summary?: SchemaSummary,
): SchemaNormalization => {
if (!isObject(value)) return { schema: value, changed: false };
if (summary !== undefined && summary.levels <= 0 && hasNestedSchema(value)) {
// A summary can be longer than the schema it replaces, such as a nullable
// array presented as anyOf. Keep the complete schema then.
const summarized = summarizedSchema(value);
return serializedBytes(summarized) < serializedBytes(value)
? { schema: summarized, changed: true }
: normalizeAdvertisedSchema(value);
}
const nested = (key: string): SchemaSummary | undefined =>
summary !== undefined && instanceChildKeys.has(key)
? { levels: summary.levels - 1 }
: summary;
const node: Record<string, unknown> = { ...value };
let changed = false;
const normalizeMap = (map: Readonly<Record<string, unknown>>) =>
Object.fromEntries(
const normalizeMap = (
key: string,
map: Readonly<Record<string, unknown>>,
) => {
const complete: [string, Readonly<Record<string, unknown>>][] = [];
return Object.fromEntries(
Object.entries(map).map(([name, child]) => {
const result = normalizeAdvertisedSchema(child);
const shared =
summary !== undefined && key === "properties"
? sharedSiblingSchema(name, child, complete)
: undefined;
if (shared !== undefined) {
changed = true;
return [name, shared];
}
const result = normalizeAdvertisedSchema(child, nested(key));
changed ||= result.changed;
return [name, result.schema];
}),
);
const normalizeList = (list: readonly unknown[]) =>
};
const normalizeList = (key: string, list: readonly unknown[]) =>
list.map((child) => {
const result = normalizeAdvertisedSchema(child);
const result = normalizeAdvertisedSchema(child, nested(key));
changed ||= result.changed;
return result;
});
for (const [key, child] of Object.entries(value)) {
if (schemaMaps.has(key) && isObject(child)) {
node[key] = normalizeMap(child);
node[key] = normalizeMap(key, child);
} else if (
schemaArrays.has(key) &&
key !== "anyOf" &&
key !== "oneOf" &&
Array.isArray(child)
) {
node[key] = normalizeList(child).map((result) => result.schema);
node[key] = normalizeList(key, child).map((result) => result.schema);
} else if (
schemaChildren.has(key) &&
key !== "not" &&
@@ -220,8 +528,8 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
key !== "else"
) {
const results = Array.isArray(child)
? normalizeList(child)
: [normalizeAdvertisedSchema(child)];
? normalizeList(key, child)
: [normalizeAdvertisedSchema(child, nested(key))];
const nestedChanged = results.some((result) => result.changed);
if (
key === "contains" &&
@@ -241,40 +549,10 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
}
}
if (typeof node.const === "boolean") {
delete node.const;
if (node.type === undefined) node.type = "boolean";
changed = true;
}
if (node.const === "") {
delete node.const;
if (node.type === undefined) node.type = "string";
changed = true;
}
if (
Array.isArray(node.enum) &&
node.enum.length > 0 &&
node.enum.every((item) => typeof item === "boolean")
) {
delete node.enum;
if (node.type === undefined) node.type = "boolean";
changed = true;
}
if (Array.isArray(node.enum) && node.enum.includes("")) {
// Removing only the empty member would reject a canonical-valid input.
// Gemini transports turn const into enum and refuse empty enum members.
const values = node.enum;
delete node.enum;
if (
node.type === undefined &&
values.every((item) => typeof item === "string")
)
node.type = "string";
changed = true;
}
if (relaxUnsupportedLiterals(node)) changed = true;
const negated = Object.hasOwn(value, "not")
? normalizeAdvertisedSchema(value.not)
? normalizeAdvertisedSchema(value.not, summary)
: undefined;
if (negated?.changed) {
// Widening a predicate under `not` narrows the containing schema.
@@ -285,13 +563,13 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
}
const conditional = Object.hasOwn(value, "if")
? normalizeAdvertisedSchema(value.if)
? normalizeAdvertisedSchema(value.if, summary)
: undefined;
const thenBranch = Object.hasOwn(value, "then")
? normalizeAdvertisedSchema(value.then)
? normalizeAdvertisedSchema(value.then, summary)
: undefined;
const elseBranch = Object.hasOwn(value, "else")
? normalizeAdvertisedSchema(value.else)
? normalizeAdvertisedSchema(value.else, summary)
: undefined;
if (conditional?.changed) {
// Changing the predicate can route canonical-valid inputs into a rejecting branch.
@@ -314,7 +592,7 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
for (const combinator of ["anyOf", "oneOf"] as const) {
const branches = value[combinator];
if (!Array.isArray(branches)) continue;
const normalized = normalizeList(branches);
const normalized = normalizeList(combinator, branches);
const schemas = normalized.map((result) => result.schema);
const branchesChanged = normalized.some((result) => result.changed);
if (combinator === "oneOf" && branchesChanged) {
@@ -353,6 +631,13 @@ const normalizeAdvertisedSchema = (value: unknown): SchemaNormalization => {
changed ||= branchesChanged;
}
if (changed) {
// A relaxed or summarized applicator can stop evaluating properties or
// items, which these keywords would then reject.
delete node.unevaluatedProperties;
delete node.unevaluatedItems;
}
return { schema: node, changed };
};
@@ -623,3 +908,95 @@ export const presentInputJsonSchema = (
description: description.join(" "),
}).schema as Record<string, unknown>;
};
/**
* Claude Desktop drops a local connector tool whose input schema, serialized
* as compact JSON, exceeds 64 KiB (#4081). Other clients take the same
* catalog, so every advertised input schema fits this limit.
*/
export const ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES = 64 * 1024;
const SUMMARIZED_GUIDANCE =
"Deeply nested and repeated inputs are advertised in summarized form to fit client schema-size limits; the server still validates complete canonical inputs.";
// The compact profile starts from the complete self-contained projection,
// not from a fitted summary of it.
const completeProjections = new WeakMap<object, Record<string, unknown>>();
/**
* Return the input schema clients receive for the selected profile: the
* fitted full advertisement, or the compact presentation of its complete
* projection.
*/
export const presentAdvertisedInputProfile = (
advertised: Record<string, unknown>,
compactPresentation: CompactInputSchemaPresentation | undefined,
): Record<string, unknown> =>
compactPresentation === undefined
? advertised
: presentAdvertisedInputSchema(
completeProjections.get(advertised) ?? advertised,
compactPresentation,
);
/**
* Fit an advertised input schema within the client size limit. A schema that
* already fits is returned unchanged. Otherwise identical sibling properties
* point at the first one, and nested schemas below the deepest level that
* still fits are summarized as their accepted types and guidance. Every
* rewrite widens the advertisement, so canonical inputs and examples remain
* accepted; the registered canonical schema still validates every call.
*/
export const fitAdvertisedInputSchema = (
schema: Record<string, unknown>,
): Record<string, unknown> => {
if (serializedBytes(schema) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES)
return schema;
// Summaries remove subschemas that a local reference could target.
const references = new JsonSchemaReferences(schema).references();
if (references.next().done !== true) return fitByCompaction(schema);
const described = {
...schema,
description: [
typeof schema.description === "string" ? schema.description : undefined,
SUMMARIZED_GUIDANCE,
]
.filter((text) => text !== undefined)
.join(" "),
};
// Size need not grow with the retained depth: a deeper candidate can be
// smaller than a summary. Keep the deepest candidate that fits.
for (let levels = schemaLevels(schema); levels >= 1; levels -= 1) {
const candidate = summarizedInputSchemaCandidate(described, levels);
if (serializedBytes(candidate) <= ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES) {
completeProjections.set(candidate, schema);
return candidate;
}
}
return fitByCompaction(schema);
};
/**
* Advertise identical siblings as pointers and summarize schemas below the
* given instance level. Exposed so tests can compare fitting candidates.
*/
export const summarizedInputSchemaCandidate = (
schema: Record<string, unknown>,
levels: number,
): Record<string, unknown> =>
normalizeAdvertisedSchema(schema, { levels }).schema as Record<
string,
unknown
>;
/** Fall back to the compact projection, which fits any budget. */
const fitByCompaction = (
schema: Record<string, unknown>,
): Record<string, unknown> => {
const { schema: fitted } = compactAdvertisedInputSchema(
schema,
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
);
completeProjections.set(fitted, schema);
return fitted;
};
+2 -2
View File
@@ -49,8 +49,8 @@ export const OFFICIAL_TOOL_CONTRACTS = [
),
official(
"inspect_native_load_image",
"Verify the provider's loaded DOS MZ or explicitly selected COM image against its immutable target snapshot: complete original/modified source-byte digests, header/module file mappings and memory digests, relocations and external entry. Returns measured observations and independent checks with mismatch coordinates. COM verification includes its imposed entry register context and does not claim PSP/runtime coverage. Overlay and uninitialized coverage remain explicit; other formats return unsupported. Does not execute or modify the target.",
z.strictObject({}),
"Inspect the provider's measured source digests, memory mappings, relocations, entry points and entry register context. Independent snapshot verification supports DOS MZ, explicitly selected COM, raw binaries and GameCube DOL; other formats retain their observations with unsupported verification. Overlay and uninitialized coverage remain explicit. Select detail summary for verification state and counts, retaining complete Evidence for inspect_analysis_view kind native-load-image: collection source_files, mappings, relocations, entry_points, entry_context, checks or sections, with indices, fields and offset/limit paging. Relocation bytes and check expected/observed payloads require explicit fields; defaults keep metadata. Null collections are unrecorded or inapplicable. Does not execute or modify the target.",
z.strictObject({ detail: analysisDetailSchema }),
),
official(
"inspect_native_data_type",
+5 -1
View File
@@ -5,6 +5,7 @@ import {
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
artifactInventorySummaryViewResultSchema,
interfaceBuilderSummaryViewResultSchema,
machoSummaryViewResultSchema,
@@ -130,7 +131,10 @@ const bookmarkFacetSchema = z.discriminatedUnion("state", [
export const officialOutputSchemas: Readonly<Record<string, z.ZodObject>> = {
search_bytes: evidenceResultOf(byteSearchResultSchema),
annotate_native_function: evidenceResultOf(nativeFunctionAnnotationsSchema),
inspect_native_load_image: evidenceResultOf(nativeLoadImageSchema),
inspect_native_load_image: analysisDetailOutputSchemaOf(
nativeLoadImageSchema,
nativeLoadImageSummaryViewResultSchema,
),
inspect_native_data_type: evidenceResultOf(nativeDataTypeSchema),
inspect_native_instruction: evidenceResultOf(nativeInstructionSchema),
resolve_native_call_targets: evidenceResultOf(nativeCallTargetsSchema),
+9 -6
View File
@@ -6,7 +6,10 @@ import { z } from "zod";
import type { ToolContract } from "./toolContractTypes.js";
import { advertisedInputExamples } from "./advertisedInputExamples.js";
import { presentInputJsonSchema } from "./inputSchemaPresentation.js";
import {
fitAdvertisedInputSchema,
presentInputJsonSchema,
} from "./inputSchemaPresentation.js";
import {
schemaArrays,
schemaChildren,
@@ -75,13 +78,13 @@ export const toolInputSchemaWithMetadata = <Contract extends ToolContract>(
libraryOptions: { ...options.libraryOptions, reused: "inline" },
});
const examples = advertisedInputExamples(contract);
return {
// Keep advertised parameters self-contained for clients that do not
// resolve local references. The registered canonical schema remains
// responsible for runtime validation.
// Keep advertised parameters self-contained for clients that do not
// resolve local references, within the per-tool client size limit. The
// registered canonical schema remains responsible for runtime validation.
return fitAdvertisedInputSchema({
...presentInputJsonSchema(inline, fallbackPropertyDescription),
...(examples.length === 0 ? {} : { examples }),
};
});
},
);
+31 -3
View File
@@ -111,10 +111,10 @@ import {
import {
nativeInventoryViewRequestSchema,
nativeInventoryViewRowSchema,
nativeInventoryViewContext,
nativeInventorySummarySchema,
projectNativeInventoryView,
} from "./nativeInventoryView.js";
import { nativeViewSourceContext } from "./nativeViewContext.js";
import {
NATIVE_INVENTORY_OPERATIONS,
nativeInventoryOperationSchema,
@@ -129,6 +129,12 @@ import {
browserScenarioViewRequestSchema,
} from "./browserScenarioViewSchemas.js";
import { projectBrowserScenarioView } from "./browserScenarioView.js";
import {
nativeLoadImageViewRequestSchema,
nativeLoadImageViewItemsSchema,
nativeLoadImageSummarySchema,
projectNativeLoadImageView,
} from "./nativeLoadImageView.js";
import {
desktopViewRequestSchema,
desktopViewItemsSchema,
@@ -387,6 +393,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
javascriptFactsViewRequestSchema,
nativeViewSchema,
nativeInventoryViewRequestSchema,
nativeLoadImageViewRequestSchema,
artifactInventoryViewRequestSchema,
wasmViewRequestSchema,
machoViewRequestSchema,
@@ -839,12 +846,21 @@ export const il2cppMetadataSummaryViewResultSchema = z.strictObject({
/** Exact inventory summary with the original observation's provider and query. */
export const nativeInventorySummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
...nativeInventoryViewContext,
...nativeViewSourceContext,
parent_operation: nativeInventoryOperationSchema,
artifact: artifactSchema.nullable(),
summary: nativeInventorySummarySchema,
});
/** Compact provider load observations retain the independent verification state. */
export const nativeLoadImageSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
...nativeViewSourceContext,
artifact: artifactSchema.nullable(),
parent_operation: z.literal("inspect_native_load_image"),
summary: nativeLoadImageSummarySchema,
});
/** Compact native matching discovery retains the complete matching report. */
export const nativeFunctionMatchingSummaryViewResultSchema = z.strictObject({
...summaryViewResultBase,
@@ -1047,6 +1063,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
modelArtifactSummaryViewResultSchema,
gpuCodeObjectsSummaryViewResultSchema,
nativeInventorySummaryViewResultSchema,
nativeLoadImageSummaryViewResultSchema,
nativeFunctionMatchingSummaryViewResultSchema,
nativeFunctionSummaryViewResultSchema,
sourceToBundleSummaryViewResultSchema,
@@ -1071,13 +1088,23 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
}),
z.strictObject({
...viewResultBase,
...nativeInventoryViewContext,
...nativeViewSourceContext,
kind: z.literal("native-inventory"),
parent_operation: nativeInventoryOperationSchema,
artifact: artifactSchema.nullable(),
view: nativeInventoryViewRequestSchema,
rows: z.array(nativeInventoryViewRowSchema),
}),
z.strictObject({
...viewResultBase,
kind: z.literal("native-load-image"),
...nativeViewSourceContext,
verification_status: z.enum(["unsupported", "verified", "mismatch"]),
parent_operation: z.literal("inspect_native_load_image"),
artifact: artifactSchema.nullable(),
view: nativeLoadImageViewRequestSchema,
items: nativeLoadImageViewItemsSchema,
}),
z.strictObject({
kind: z.literal("protobuf"),
view: protobufViewRequestSchema,
@@ -1338,6 +1365,7 @@ const retainedViewProjectors = new Map<string, typeof projectWasmArtifactView>([
["inspect_erlang_beam", projectErlangBeamView],
["inspect_devicetree", projectDevicetreeView],
["inspect_native_exports", projectNativeExportsView],
["inspect_native_load_image", projectNativeLoadImageView],
["inspect_python_pyc", projectPythonPycView],
["frida_session_status", projectFridaSessionStatusView],
["gdb_console", projectCommandObservationView],
@@ -7,8 +7,6 @@ import {
isNativeInventoryOperation,
nativeInventoryRowsSchemas,
} from "../native/nativeInventory.js";
import { digestSchema } from "../digests.js";
import { providerSchema } from "../evidence.js";
import { err, ok, type Result } from "../result.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import type {
@@ -53,12 +51,6 @@ export const nativeInventoryViewRowSchema = z.strictObject({
record: inventoryRecordSchema.partial().required({ address: true }),
});
export const nativeInventoryViewContext = {
source_provider: providerSchema.nullable(),
source_analysis_profile_digest: digestSchema.nullable(),
source_parameters: jsonValueSchema,
};
export const nativeInventorySummarySchema = z.strictObject({
collection: z.enum(["procedures", "strings", "names"]),
row_count: z.number().int().nonnegative(),
@@ -0,0 +1,301 @@
import { z } from "zod";
import {
nativeLoadImageCheckSchema,
nativeLoadImageObservationSchema,
nativeLoadImageSchema,
nativeUnsupportedLoadImageSchema,
nativeDosLoadImageSchema,
nativeRawLoadImageSchema,
nativeDolLoadImageSchema,
} from "../native/nativeLoadImage.js";
import { nativeAddressObservationSchema } from "../native/nativeAddress.js";
import { jsonValueSchema } from "../jsonValue.js";
import { err, ok, type Result } from "../result.js";
import type { AnalysisError } from "../analysisErrorBase.js";
import type {
AnalysisViewParent,
AnalysisViewRequest,
UnsignedAnalysisView,
} from "./analysisView.js";
import { analysisViewInputError } from "./analysisViewErrors.js";
import { createAnalysisViewResultReader } from "./analysisViewParent.js";
import {
boundedViewMessages,
completeWithinViewCoverage,
pageViewCoverage,
} from "./analysisViewCoverage.js";
const observation = nativeLoadImageObservationSchema.shape;
const rows = {
source_files: observation.source_files.element,
mappings: observation.mappings.element,
relocations: observation.relocations.element,
entry_points: z.strictObject({ address: nativeAddressObservationSchema }),
entry_context: observation.entry_context.element,
checks: nativeLoadImageCheckSchema,
sections: nativeDolLoadImageSchema.shape.sections.element,
};
const defaultFields: Partial<Record<keyof typeof rows, readonly string[]>> = {
relocations: rows.relocations
.pick({
address: true,
segment: true,
offset: true,
status: true,
type: true,
values: true,
})
.keyof().options,
checks: rows.checks
.pick({ name: true, matched: true, file_offset: true, address: true })
.keyof().options,
};
const selection = {
kind: z.literal("native-load-image"),
offset: z.int().nonnegative().default(0),
limit: z.int().positive().default(64),
indices: z.array(z.int().nonnegative()).optional(),
};
const request = <Name extends keyof typeof rows>(name: Name) =>
z.strictObject({
...selection,
collection: z.literal(name),
fields: z
.array(rows[name].keyof())
.optional()
.describe(
name === "relocations"
? "Defaults to address, segment, offset, status, type and values. Select original_bytes_hex or memory_bytes_hex explicitly."
: name === "checks"
? "Defaults to name, matched, file_offset and address. Select expected or observed explicitly; these can contain complete comparison inventories."
: "Omit to return every recorded field in each selected row.",
),
});
/** Pages retain producer order and original indices within the parent Evidence. */
export const nativeLoadImageViewRequestSchema = z.discriminatedUnion(
"collection",
[
request("source_files"),
request("mappings"),
request("relocations"),
request("entry_points"),
request("entry_context"),
request("checks"),
request("sections"),
],
);
const items = <Name extends keyof typeof rows>(name: Name) => {
const records = z.array(
z.strictObject({
source_index: z.int().nonnegative(),
record: rows[name].partial(),
}),
);
return z.strictObject({
collection: z.literal(name),
rows:
name === "checks" || name === "sections" ? records.nullable() : records,
});
};
export const nativeLoadImageViewItemsSchema = z.discriminatedUnion(
"collection",
[
items("source_files"),
items("mappings"),
items("relocations"),
items("entry_points"),
items("entry_context"),
items("checks"),
items("sections"),
],
);
// Keep the format owner's verification states and small identity facts. A
// measured unsupported format must never become a verified empty check set.
export const nativeLoadImageSummarySchema = z.strictObject({
verification: z.union([
nativeUnsupportedLoadImageSchema.omit({
observations: true,
limitations: true,
}),
nativeDosLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
}),
nativeRawLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
}),
nativeDolLoadImageSchema.omit({
observations: true,
limitations: true,
checks: true,
sections: true,
}),
]),
observation: nativeLoadImageObservationSchema.pick({
executable_format: true,
language_id: true,
compiler_spec_id: true,
image_base: true,
default_address_space: true,
}),
counts: z.strictObject({
source_files: z.int().nonnegative(),
mappings: z.int().nonnegative(),
relocations: z.int().nonnegative(),
entry_points: z.int().nonnegative(),
entry_context: z.int().nonnegative(),
metadata_recovery: z.int().nonnegative().nullable(),
checks: z.int().nonnegative().nullable(),
sections: z.int().nonnegative().nullable(),
}),
});
const readReport = createAnalysisViewResultReader(nativeLoadImageSchema);
type Row = z.output<(typeof rows)[keyof typeof rows]>;
type PageRequest = z.output<typeof nativeLoadImageViewRequestSchema>;
/** Exact source indices need no inventory scan or persistent lookup index. */
const selectRows = (source: readonly (Row | string)[], view: PageRequest) => {
const selectedFields = view.fields ?? defaultFields[view.collection];
const fields =
selectedFields === undefined ? undefined : new Set<string>(selectedFields);
const project = (value: Row | string, source_index: number) => {
const record = typeof value === "string" ? { address: value } : value;
return {
source_index,
record:
fields === undefined
? record
: Object.fromEntries(
Object.entries(record).filter(([key]) => fields.has(key)),
),
};
};
if (view.indices === undefined)
return {
total: source.length,
rows: source
.slice(view.offset, view.offset + view.limit)
.map((value, index) => project(value, view.offset + index)),
};
const indices = [...new Set(view.indices)]
.filter((index) => index < source.length)
.sort((left, right) => left - right);
return {
total: indices.length,
// The format owner validated a dense collection; these indices are in range.
rows: indices
.slice(view.offset, view.offset + view.limit)
.map((index) => project(source[index]!, index)),
};
};
/** Query the owner's parsed historical observations without reopening the target. */
export const projectNativeLoadImageView = (
parent: AnalysisViewParent,
view: AnalysisViewRequest,
): Result<UnsignedAnalysisView, AnalysisError> => {
if (view.kind !== "summary" && view.kind !== "native-load-image")
return err(
analysisViewInputError([
{
path: ["view"],
reason: "invalid_value",
message:
"Native load-image Evidence supports summary and native-load-image views.",
},
]),
);
const parsed = readReport(parent);
if (!parsed.ok) return parsed;
const report = parsed.value;
const observed = report.observations;
const { observations: _, limitations, ...verification } = report;
const {
checks: __,
sections: ___,
...smallVerification
} = {
checks: undefined,
sections: undefined,
...verification,
};
const {
source_files,
mappings,
relocations,
entry_points,
entry_context,
metadata_recovery,
...identity
} = observed;
const shared = {
parent_evidence_id: parent.evidenceId,
parent_operation: "inspect_native_load_image" as const,
parent_digest: parent.evidenceId.slice(3),
artifact: parent.artifact ?? null,
source_provider: parent.provider ?? null,
source_analysis_profile_digest: parent.analysisProfileDigest ?? null,
source_parameters: jsonValueSchema.parse(parent.parameters ?? null),
limitations: boundedViewMessages(
[...parent.limitations, ...limitations],
parent.evidenceId,
"limitations",
),
unknowns: [
"Counts and pages describe retained provider observations, not complete runtime mappings or relocation behavior.",
"Null collection counts or rows mean not recorded or not applicable, not observed absence.",
...(report.status === "unsupported" ? [report.reason] : []),
],
};
if (view.kind === "summary")
return ok({
...shared,
kind: "summary",
view,
summary: {
verification: smallVerification,
observation: identity,
counts: {
source_files: source_files.length,
mappings: mappings.length,
relocations: relocations.length,
entry_points: entry_points.length,
entry_context: entry_context.length,
metadata_recovery: metadata_recovery?.length ?? null,
checks: "checks" in report ? report.checks.length : null,
sections: "sections" in report ? report.sections.length : null,
},
},
coverage: completeWithinViewCoverage(1, 1),
});
const selected =
view.collection === "checks"
? "checks" in report
? report.checks
: undefined
: view.collection === "sections"
? "sections" in report
? report.sections
: undefined
: observed[view.collection];
const page = selectRows(selected ?? [], view);
return ok({
...shared,
kind: "native-load-image",
verification_status: report.status,
view,
items: nativeLoadImageViewItemsSchema.parse({
collection: view.collection,
rows: selected === undefined ? null : page.rows,
}),
coverage: pageViewCoverage(view.offset, page.rows.length, page.total),
});
};
@@ -0,0 +1,10 @@
import { digestSchema } from "../digests.js";
import { providerSchema } from "../evidence.js";
import { jsonValueSchema } from "../jsonValue.js";
/** Native projections reference the original provider/profile without repeating its full commitment. */
export const nativeViewSourceContext = {
source_provider: providerSchema.nullable(),
source_analysis_profile_digest: digestSchema.nullable(),
source_parameters: jsonValueSchema,
};
+68 -62
View File
@@ -84,74 +84,80 @@ export const nativeLoadImageCheckSchema = z.strictObject({
address: nativeAddressObservationSchema.nullable(),
});
/** Read-only native image verification; verified applies only to declared coverage. */
export const nativeLoadImageSchema = z.union([
z.strictObject({
status: z.literal("unsupported"),
reason: z.string().min(1),
observations: nativeLoadImageObservationSchema,
limitations: z.array(z.string().min(1)),
/** Measured observations when independent format verification is unsupported. */
export const nativeUnsupportedLoadImageSchema = z.strictObject({
status: z.literal("unsupported"),
reason: z.string().min(1),
observations: nativeLoadImageObservationSchema,
limitations: z.array(z.string().min(1)),
});
export const nativeDosLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.enum(["dos-mz", "dos-com"]),
target_sha256: digest,
load_segment: integer.max(0xffff),
header_bytes: integer,
module_bytes: integer,
overlay_bytes: integer,
entry: z.strictObject({
relative_segment: integer.max(0xffff),
offset: integer.max(0xffff),
linear_address: nativeAddressObservationSchema,
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.enum(["dos-mz", "dos-com"]),
target_sha256: digest,
load_segment: integer.max(0xffff),
header_bytes: integer,
module_bytes: integer,
overlay_bytes: integer,
entry: z.strictObject({
relative_segment: integer.max(0xffff),
offset: integer.max(0xffff),
linear_address: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
export const nativeRawLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("raw-binary"),
target_sha256: digest,
architecture: binaryArchitectureSchema,
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
base_address: nativeAddressObservationSchema,
file_bytes: integer.min(1),
entry: z.strictObject({
address: nativeAddressObservationSchema,
file_offset: z.literal(0),
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("raw-binary"),
target_sha256: digest,
architecture: binaryArchitectureSchema,
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
base_address: nativeAddressObservationSchema,
file_bytes: integer.min(1),
entry: z.strictObject({
address: nativeAddressObservationSchema,
file_offset: z.literal(0),
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
}),
z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("gamecube-dol"),
target_sha256: digest,
architecture: z.literal("powerpc"),
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
file_bytes: integer.min(1),
entry: z.strictObject({ address: nativeAddressObservationSchema }),
sections: z.array(
z.strictObject({
kind: z.enum(["text", "data"]),
index: integer,
file_offset: nativeAddressObservationSchema,
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
),
bss: z.strictObject({
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
export const nativeDolLoadImageSchema = z.strictObject({
status: z.enum(["verified", "mismatch"]),
format: z.literal("gamecube-dol"),
target_sha256: digest,
architecture: z.literal("powerpc"),
language_id: z.string().min(1),
compiler_spec_id: z.string().min(1),
file_bytes: integer.min(1),
entry: z.strictObject({ address: nativeAddressObservationSchema }),
sections: z.array(
z.strictObject({
kind: z.enum(["text", "data"]),
index: integer,
file_offset: nativeAddressObservationSchema,
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
),
bss: z.strictObject({
address: nativeAddressObservationSchema,
size: nativeAddressObservationSchema,
}),
observations: nativeLoadImageObservationSchema,
checks: z.array(nativeLoadImageCheckSchema),
limitations: z.array(z.string().min(1)),
});
/** Read-only native image verification; verified applies only to declared coverage. */
export const nativeLoadImageSchema = z.union([
nativeUnsupportedLoadImageSchema,
nativeDosLoadImageSchema,
nativeRawLoadImageSchema,
nativeDolLoadImageSchema,
]);
/** Provider-neutral native load-image result. */
+17 -3
View File
@@ -54,6 +54,7 @@ import {
ghidraFailureOutput,
GHIDRA_JVM_OOME_MARKER,
GHIDRA_PROCESS_DIAGNOSTIC_BYTES,
type GhidraDiagnosticsScope,
} from "./GhidraDiagnostics.js";
import { GhidraLaunchError, type GhidraLaunch } from "./GhidraLauncher.js";
import { GhidraResponseRouter } from "./GhidraResponseRouter.js";
@@ -158,7 +159,7 @@ export class GhidraClient {
#startupAttempt: GhidraStartupAttempt | undefined;
#closePromise: Promise<void> | undefined;
readonly #failure = bindGhidraSessionFailure(
() => this.#diagnostics(),
(scope) => this.#diagnostics(scope),
(value, incomplete) => this.#redactAuthentication(value, incomplete),
);
readonly #wire: GhidraWire;
@@ -1339,7 +1340,17 @@ export class GhidraClient {
});
}
#diagnostics(): Readonly<Record<string, JsonValue>> {
#diagnostics(
requestedScope: GhidraDiagnosticsScope = "session",
): Readonly<Record<string, JsonValue>> {
// Startup errors still need launcher evidence. A request projection must
// have both a completed handshake and its own active output checkpoint.
const scope =
requestedScope === "request" &&
this.#startupSucceeded() &&
this.#activeOutputCheckpoint !== undefined
? "request"
: "session";
const currentSnapshot = this.#process?.snapshot() ?? this.#processSnapshot;
const snapshot: ProviderProcessSnapshot | undefined =
this.#outputDrainIncomplete && currentSnapshot !== undefined
@@ -1350,7 +1361,9 @@ export class GhidraClient {
}
: currentSnapshot;
const checkpoint =
this.#activeOutputCheckpoint ?? this.#startupOutputCheckpoint;
scope === "request"
? this.#activeOutputCheckpoint
: (this.#activeOutputCheckpoint ?? this.#startupOutputCheckpoint);
const scoped =
checkpoint === undefined
? undefined
@@ -1360,6 +1373,7 @@ export class GhidraClient {
? undefined
: ghidraFailureOutput(scoped, this.#token, this.#outputDrainIncomplete);
return createGhidraDiagnostics({
scope,
targetPath: this.#options.targetPath,
targetSha256: this.#options.targetSha256,
...(this.#targetAdmission === undefined
+58
View File
@@ -5,6 +5,64 @@ import {
} from "./GhidraDiagnostics.js";
import { expect, it } from "vitest";
it("keeps request output and provenance without projecting launcher history", () => {
const output = { text: "launcher output", bytes: 15, observedBytes: 15 };
const options = {
targetPath: "./caller-selected/../target",
targetSha256: "a".repeat(64),
transport: "unix-socket" as const,
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
runtimeRoot: "/owned/runtime",
snapshot: { stdout: output, stderr: output, exitCode: null, signal: null },
previous: {},
failure_output: { stdout: { text: "query output" }, complete: false },
};
const request = createGhidraDiagnostics({ ...options, scope: "request" });
expect(request).toMatchObject({
diagnostic_scope: "request",
target_path: options.targetPath,
target_sha256: options.targetSha256,
profile_digest: options.profileDigest,
runtime_root: options.runtimeRoot,
failure_output: options.failure_output,
});
expect(request).not.toHaveProperty("stdout");
expect(request).not.toHaveProperty("stderr");
expect(createGhidraDiagnostics(options)).toMatchObject({
stdout: { text: output.text },
stderr: { text: output.text },
failure_output: options.failure_output,
});
});
it("keeps cached session output owned while projecting historical request coordinates", () => {
const previous = {
target_path: "/selected/target",
ghidra_log_path: "/owned/project/ghidra.log",
stdout: { text: "old launcher output" },
stderr: { text: "old launcher stderr" },
launch_support: [{ stdout: "JDK version" }],
failure_output: { stdout: { text: "old request" }, complete: true },
};
const request = createGhidraDiagnostics({
scope: "request",
targetPath: "/selected/target",
targetSha256: "a".repeat(64),
transport: "unix-socket",
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
previous,
});
expect(request).toEqual({
diagnostic_scope: "request",
target_path: previous.target_path,
ghidra_log_path: previous.ghidra_log_path,
});
expect(previous.stdout.text).toBe("old launcher output");
expect(previous.failure_output.stdout.text).toBe("old request");
});
it("does not reuse historical operation-scoped failure output without a current owner", () => {
const diagnostics = createGhidraDiagnostics({
targetPath: "/tmp/target",
+25 -9
View File
@@ -13,8 +13,12 @@ export const GHIDRA_PROCESS_DIAGNOSTIC_BYTES = 8 * 1024 * 1024;
/** JVM error class located in all launcher output, beyond the retention budget. */
export const GHIDRA_JVM_OOME_MARKER = "java.lang.OutOfMemoryError";
/** A rejected caller input owns request output, not the launcher's history. */
export type GhidraDiagnosticsScope = "session" | "request";
/** Inputs needed to project bounded diagnostics for one Ghidra runtime. */
export interface GhidraDiagnosticsOptions {
readonly scope?: GhidraDiagnosticsScope;
readonly targetPath: string;
readonly targetSha256: string;
readonly targetAdmission?: JsonValue;
@@ -43,9 +47,16 @@ export const createGhidraDiagnostics = (
) {
const history: Record<string, JsonValue> = { ...options.previous };
delete history.failure_output;
if (options.scope === "request") {
delete history.stdout;
delete history.stderr;
delete history.launch_support;
history.diagnostic_scope = "request";
}
return history;
}
return {
...(options.scope === "request" ? { diagnostic_scope: "request" } : {}),
target_path: options.targetPath,
target_sha256: options.targetSha256,
...(options.targetAdmission === undefined
@@ -71,7 +82,8 @@ export const createGhidraDiagnostics = (
ghidra_log_path: options.launch.ghidraLogPath,
script_log_path: options.launch.scriptLogPath,
process_id: options.launch.process.pid ?? null,
...(options.launch.launchSupport === undefined
...(options.scope === "request" ||
options.launch.launchSupport === undefined
? {}
: {
launch_support: options.launch.launchSupport.map((output) => ({
@@ -90,14 +102,18 @@ export const createGhidraDiagnostics = (
stop_requested: options.snapshot.stopRequested === true,
diagnostic_truncated: options.snapshot.diagnosticTruncated === true,
output_incomplete: options.snapshot.outputIncomplete === true,
stdout: ghidraDiagnosticStream(
options.snapshot.stdout,
options.token,
),
stderr: ghidraDiagnosticStream(
options.snapshot.stderr,
options.token,
),
...(options.scope === "request"
? {}
: {
stdout: ghidraDiagnosticStream(
options.snapshot.stdout,
options.token,
),
stderr: ghidraDiagnosticStream(
options.snapshot.stderr,
options.token,
),
}),
}),
...(options.failure_output === undefined
? {}
+2 -6
View File
@@ -99,6 +99,7 @@ import {
} from "./GhidraProviderCapabilities.js";
import {
GHIDRA_STARTUP_DEADLINE_ELAPSED,
isGhidraInputFailureCode,
type GhidraSessionError,
} from "./GhidraSessionError.js";
import type { GhidraSessionInfo } from "./GhidraSessionValues.js";
@@ -1046,12 +1047,7 @@ const projectPrimarySessionError = (
"Retry this search in literal mode or simplify the regex.",
},
);
if (
failure.kind === "remote" &&
["invalid_request", "not_found", "ambiguous"].includes(
failure.remoteCode ?? "",
)
)
if (failure.kind === "remote" && isGhidraInputFailureCode(failure.remoteCode))
return new AnalysisInputError(operation, context, [
{ path: [], reason: "invalid_value", message: failure.message },
]);
+33
View File
@@ -7,6 +7,39 @@ import {
bindGhidraSessionFailure,
} from "./GhidraSessionError.js";
import { GhidraLaunchError } from "./GhidraLauncher.js";
import { createGhidraDiagnostics } from "./GhidraDiagnostics.js";
it.each(["decompile_failed", "future_remote_failure"])(
"retains complete session diagnostics for non-input remote code %s",
(remoteCode) => {
const factory = bindGhidraSessionFailure((scope) =>
createGhidraDiagnostics({
scope,
targetPath: "/selected/target",
targetSha256: "a".repeat(64),
transport: "unix-socket",
providerVersion: "12.1.4",
profileDigest: "b".repeat(64),
runtimeRoot: "/owned/runtime",
snapshot: {
stdout: { text: "engine diagnostic", bytes: 17, observedBytes: 17 },
stderr: { text: "", bytes: 0, observedBytes: 0 },
exitCode: null,
signal: null,
},
previous: {},
}),
);
const failure = factory("remote", "Engine failed", undefined, {
remoteCode,
});
expect(failure.diagnostics).toMatchObject({
stdout: { text: "engine diagnostic" },
remote_code: remoteCode,
});
expect(failure.diagnostics).not.toHaveProperty("diagnostic_scope");
},
);
it("reports truncated cyclic causes in serializable diagnostics", () => {
const cause = new Error("cyclic launcher failure");
+16 -3
View File
@@ -6,6 +6,7 @@ import {
import type { JsonValue } from "../domain/jsonValue.js";
import { DarwinProcessOwnershipInspectionError } from "../process/DarwinProcessRunTokenReader.js";
import { boundedUtf8Text } from "../process/boundedUtf8Text.js";
import type { GhidraDiagnosticsScope } from "./GhidraDiagnostics.js";
import { GhidraLaunchSupportError } from "./GhidraJavaLaunch.js";
import {
GhidraLaunchError,
@@ -184,10 +185,16 @@ const createGhidraSessionError = (failure: {
},
);
/** Bind session-failure construction to a live, token-redacted diagnostic view. */
/** Bridge input failures share classification with their domain error projection. */
export const isGhidraInputFailureCode = (code: string | undefined): boolean =>
code === "invalid_request" || code === "not_found" || code === "ambiguous";
/** Bind session-failure construction to its causal diagnostic scope. */
export const bindGhidraSessionFailure =
(
diagnostics: () => Readonly<Record<string, JsonValue>>,
diagnostics: (
scope: GhidraDiagnosticsScope,
) => Readonly<Record<string, JsonValue>>,
redact: (value: string, incomplete?: boolean) => string = (value) => value,
) =>
(
@@ -203,7 +210,13 @@ export const bindGhidraSessionFailure =
kind,
message: redact(message),
diagnostics: {
...diagnostics(),
...diagnostics(
kind === "remote" &&
(isGhidraInputFailureCode(options.remoteCode) ||
options.remoteCode === "invalid_function_name")
? "request"
: "session",
),
...(cause === undefined
? {}
: { failure_cause: describeCause(cause, redact) }),
+3 -5
View File
@@ -26,12 +26,10 @@ import {
} from "@modelcontextprotocol/server";
import { analysisErrorProjectionSchema } from "../contracts/errorSchemas.js";
import {
presentAdvertisedInputSchema,
type CompactInputSchemaPresentation,
} from "../contracts/compactInputSchema.js";
import type { CompactInputSchemaPresentation } from "../contracts/compactInputSchema.js";
import type { createCatalogIdentity } from "../catalogIdentity.js";
import { transformAdvertisedInputJsonSchema } from "../contracts/toolSchemaMetadata.js";
import { presentAdvertisedInputProfile } from "../contracts/inputSchemaPresentation.js";
import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js";
import {
AnalysisInputError,
@@ -259,7 +257,7 @@ export class EvidenceMcpServerCore extends RequestTrackingMcpServer {
typeof transformAdvertisedInputJsonSchema
>[0],
(projected) =>
presentAdvertisedInputSchema(projected, { budgetBytes: budget }),
presentAdvertisedInputProfile(projected, { budgetBytes: budget }),
)
: undefined;
const registeredConfig =
+4 -3
View File
@@ -8,9 +8,10 @@ import {
type EvidenceToolFamilyOptions,
} from "./retainedSummaryDelivery.js";
const SUMMARY_OPERATIONS: ReadonlySet<string> = new Set(
NATIVE_INVENTORY_OPERATIONS,
);
const SUMMARY_OPERATIONS: ReadonlySet<string> = new Set([
...NATIVE_INVENTORY_OPERATIONS,
"inspect_native_load_image",
]);
const withoutDetail = withoutDeliveryDetail(SUMMARY_OPERATIONS);
/** Direct tools share Evidence recording/projection with the other provider-backed tools. */
@@ -0,0 +1,79 @@
import { Cli } from "incur";
import pino from "pino";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { registerCoreNativeCommands } from "../../../src/cli/coreNativeCommands.js";
import {
createEvidence,
parseEvidence,
withEvidenceCallerInputPath,
} from "../../../src/domain/evidence.js";
import { parseEvidenceBundle } from "../../../src/domain/evidenceBundle.js";
import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js";
import { nativeDosLoadImageFixture } from "../../fixtures/nativeLoadImage.js";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
it("preserves the relative request path in a load-image CLI summary and its complete bundle", async () => {
const directory = await createTestTempDirectory("rea-load-image-cli-path-");
const destination = join(directory, "complete.json");
const selectedPath = "./alias/../selected.exe";
const parent = withEvidenceCallerInputPath(
createEvidence(
{ path: "/selected.exe", sha256: "a".repeat(64), format: "dos-mz" },
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "inspect_native_load_image",
parameters: {},
result: nativeDosLoadImageFixture(),
},
),
selectedPath,
);
const run = vi.fn(async () => parent);
const cli = Cli.create("rea-load-image-cli-test", { sync: false });
registerCoreNativeCommands(cli, pino({ enabled: false }), run);
let stdout = "";
await cli.serve(
[
"inspect-native-load-image",
selectedPath,
"--detail",
"summary",
"--evidence_output",
destination,
"--json",
],
{
env: {},
stdout: (text) => {
stdout += text;
},
exit: () => undefined,
},
);
const receipt = JSON.parse(stdout);
const summary = parseEvidence(receipt.summary);
expect(summary.caller_input_path).toBe(selectedPath);
expect(run).toHaveBeenCalledExactlyOnceWith(
selectedPath,
"inspect_native_load_image",
{},
expect.any(Object),
);
const bundle = parseEvidenceBundle(
JSON.parse(await readFile(destination, "utf8")),
);
expect(bundle.records).toEqual([parent]);
expect(summary.normalized_result).toMatchObject({
parent_evidence_id: parent.evidence_id,
});
const later = await inspectAnalysisView({
source: { kind: "inline", evidence: parent },
view: { kind: "summary" },
});
if (!later.ok) throw later.error;
expect(later.value).not.toHaveProperty("caller_input_path");
expect(later.value.evidence_id).toBe(summary.evidence_id);
});
@@ -67,8 +67,8 @@ function invalidComparisonInputs(
{ before: passive.before },
{ before: { inspection: {} }, after: passive.after },
{
before: passive.before,
after: { ...passive.after, webmcp: {} },
before: { ...passive.before, webmcp: {} },
after: passive.after,
},
{ before_scenario: {}, after_scenario: {} },
{ before_scenario: scenario.before_scenario },
@@ -146,6 +146,10 @@ describe("browser comparison input boundary", () => {
"before + after; before_scenario + after_scenario + [normalization]",
);
const runtimeOnlyInvalid = [
// The advertisement fits the 64 KiB client limit by presenting
// `after` as "Same schema as `before`" (#4081); the canonical
// parser still rejects a malformed `after` capture.
{ before: passive.before, after: { ...passive.after, webmcp: {} } },
{ ...passive, ...scenario },
{ ...passive, normalization: { rules: [] } },
{
+11 -6
View File
@@ -267,12 +267,17 @@ describe("browser scenario MCP tool", () => {
validateComparison(comparisonInput),
JSON.stringify(validateComparison.errors),
).toBe(true);
for (const malformed of [
{ before_scenario: captureResult },
{ before_scenario: captureResult, after: captureResult },
{ ...comparisonInput, before: captureResult, after: captureResult },
]) {
expect(validateFullComparison(malformed)).toBe(false);
for (const [malformed, advertisedRejects] of [
[{ before_scenario: captureResult }, true],
// The fitted advertisement summarizes `after` as "Same schema as
// `before`"; only the canonical contract rejects a scenario in it.
[{ before_scenario: captureResult, after: captureResult }, false],
[
{ ...comparisonInput, before: captureResult, after: captureResult },
true,
],
] as const) {
expect(validateFullComparison(malformed)).toBe(!advertisedRejects);
const rejected = await client.callTool({
name: "compare_web_captures",
arguments: malformed,
@@ -10,7 +10,10 @@ import {
} from "../../../src/contracts/toolContracts.js";
import { PROMPT_CONTRACTS } from "../../../src/contracts/promptContracts.js";
import { advertisedInputJsonSchema } from "../../../src/contracts/toolSchemaMetadata.js";
import { presentAdvertisedInputSchema } from "../../../src/contracts/compactInputSchema.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
presentAdvertisedInputProfile,
} from "../../../src/contracts/inputSchemaPresentation.js";
import { createCatalogDigests } from "../../../src/catalogIdentity.js";
import {
schemaArrays,
@@ -144,7 +147,7 @@ describe("MCP compact input schema profile", () => {
].entries()) {
for (const contract of TOOL_CONTRACTS) {
expect(toolsByName[index]?.get(contract.name)?.inputSchema).toEqual(
presentAdvertisedInputSchema(
presentAdvertisedInputProfile(
advertisedInputJsonSchema(contract),
presentation,
),
@@ -208,6 +211,47 @@ it("advertises every tool input schema within the provider budget", async () =>
).toEqual([]);
});
it("advertises every input schema within the client per-tool limit in both profiles", async () => {
// Claude Desktop drops a tool whose compact-JSON input schema exceeds 64 KiB.
for (const server of [fullServer(), compactServer()]) {
registerCatalog(server);
const tools = await listToolsThroughTransport(server);
expect(tools).toHaveLength(TOOL_CONTRACTS.length);
expect(
tools.flatMap(({ name, inputSchema }) =>
serializedBytes(inputSchema) > ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES
? [[name, serializedBytes(inputSchema)]]
: [],
),
).toEqual([]);
}
const captures = advertisedInputJsonSchema(
toolContract("compare_web_captures"),
);
// The complete capture schemas are each shared by two sides.
expect(captures).toMatchObject({
description: expect.stringContaining("summarized form"),
properties: {
before: {
properties: {
inspection: { type: "object", properties: expect.any(Object) },
},
},
after: {
type: "object",
description: expect.stringMatching(/ Same schema as `before`\.$/u),
},
before_scenario: { required: expect.arrayContaining(["steps"]) },
after_scenario: {
type: "object",
description: expect.stringMatching(
/ Same schema as `before_scenario`\.$/u,
),
},
},
});
});
it("keeps output schemas and tool descriptions identical to the default profile", async () => {
const compact = compactServer();
const full = fullServer();
@@ -14,6 +14,103 @@ import { functionDossierSchema } from "../../../src/domain/hopperValues.js";
import { ok } from "../../../src/domain/result.js";
import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js";
import { toolContract } from "../../../src/contracts/toolContracts.js";
import { Ajv2020 } from "ajv/dist/2020.js";
import { analysisViewResultSchema } from "../../../src/domain/analysisView/analysisView.js";
it("retains complete load-image facts and pages selected relocations through the advertised SDK contracts", async () => {
const fixture = fixtureDosOverlapLoadImage();
const directory = await createTestTempDirectory("rea-mcp-load-summary-");
const targetPath = join(directory, "inert.exe");
await writeFile(targetPath, fixture.bytes);
const harness = await connectGhidraMcp(
"ghidra-load-summary",
async () => ok(jsonValueSchema.parse(fixture.observation)),
{
targetPath,
sessionTarget: {
name: "inert.exe",
language_id: fixture.observation.language_id,
compiler_spec_id: fixture.observation.compiler_spec_id,
image_base: fixture.observation.image_base,
default_address_space: fixture.observation.default_address_space,
sha256: fixture.sha256,
},
attestDosTargetSnapshot: async (observations) =>
compareDosClientFixture(fixture.bytes, fixture.sha256, observations),
},
);
try {
const catalog = await harness.mcp.listTools();
const ajv = new Ajv2020({ strict: false, validateFormats: false });
const call = async (
name: "inspect_native_load_image" | "inspect_analysis_view",
input: Record<string, unknown>,
) => {
const definition = catalog.tools.find((tool) => tool.name === name);
if (!definition?.outputSchema) throw new Error("Missing named schema");
const inputSchema: Record<string, unknown> = definition.inputSchema;
const outputSchema: Record<string, unknown> = definition.outputSchema;
expect(ajv.validateSchema(inputSchema)).toBe(true);
expect(ajv.validateSchema(outputSchema)).toBe(true);
const reply = await harness.mcp.callTool(
{ name, arguments: input },
{ toolDefinition: definition },
);
expect(reply.isError).not.toBe(true);
expect(
ajv.validate(outputSchema, reply.structuredContent),
JSON.stringify(ajv.errors),
).toBe(true);
return sessionEvidence(harness.session, reply.structuredContent);
};
const complete = await call("inspect_native_load_image", {});
const summary = await call("inspect_native_load_image", {
detail: "summary",
});
expect(summary.normalized_result).toMatchObject({
parent_evidence_id: complete.evidence_id,
source_provider: complete.provider,
source_analysis_profile_digest: complete.analysis_profile?.digest,
source_parameters: complete.parameters,
summary: {
verification: { status: "verified", format: "dos-mz" },
counts: { relocations: 2 },
},
});
const calls = harness.calls.length;
const page = await call("inspect_analysis_view", {
source: { kind: "retained-evidence", evidence_id: complete.evidence_id },
view: {
kind: "native-load-image",
collection: "relocations",
offset: 1,
limit: 1,
fields: ["address", "memory_bytes_hex"],
},
});
const parsed = analysisViewResultSchema.parse(page.normalized_result);
if (parsed.kind !== "native-load-image")
throw new Error("Expected load-image page");
expect(parsed.verification_status).toBe("verified");
expect(parsed.items.rows).toEqual([
{
source_index: 1,
record: {
address: fixture.observation.relocations[1]!.address,
memory_bytes_hex:
fixture.observation.relocations[1]!.memory_bytes_hex,
},
},
]);
expect(harness.calls).toHaveLength(calls);
expect(harness.session.evidenceById(complete.evidence_id)).toEqual(
complete,
);
expect(complete.parameters).toEqual({});
} finally {
await harness.close();
}
});
it.each([
{ state: "final", failed: [] },
@@ -288,12 +288,14 @@ it("refuses summary before provider work when no retaining session is configured
InMemoryTransport.createLinkedPair();
await server.connect(serverTransport);
await client.connect(clientTransport);
const reply = await client.callTool({
name: "list_procedures",
arguments: { detail: "summary" },
});
expect(reply.isError).toBe(true);
expect(parseMcpToolError(reply).error.code).toBe("capability_unavailable");
for (const name of ["list_procedures", "inspect_native_load_image"]) {
const reply = await client.callTool({
name,
arguments: { detail: "summary" },
});
expect(reply.isError).toBe(true);
expect(parseMcpToolError(reply).error.code).toBe("capability_unavailable");
}
expect(calls).toBe(0);
});
@@ -0,0 +1,95 @@
import { expect, it } from "vitest";
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
import { analysisViewResultSchema } from "../../../src/domain/analysisView/analysisView.js";
import {
nativeDolLoadImageFixture,
nativeRawLoadImageFixture,
} from "../../fixtures/nativeLoadImage.js";
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
it("preserves raw and DOL verification facts and selected DOL sections through the advertised SDK output", async () => {
const { call } = await connectLocalToolsMcp();
for (const report of [
nativeRawLoadImageFixture(),
nativeDolLoadImageFixture(),
]) {
const parent = createEvidence(
{
path: "/selected/fixture",
sha256: report.target_sha256,
format: report.format,
architecture: report.architecture,
},
{ id: "ghidra", name: "Ghidra", version: "12.1.4" },
{
operation: "inspect_native_load_image",
parameters: {},
result: report,
},
);
const source = { kind: "inline", evidence: parent };
const summary = await call("inspect_analysis_view", {
source,
view: { kind: "summary" },
});
expect(summary.isError, JSON.stringify(summary.content)).not.toBe(true);
const result = analysisViewResultSchema.parse(
parseEvidence(summary.structuredContent).normalized_result,
);
const {
observations: _observations,
checks,
limitations: _limitations,
...verification
} = report;
if ("sections" in verification)
Reflect.deleteProperty(verification, "sections");
expect(result).toMatchObject({
summary: {
verification,
counts: {
checks: checks.length,
sections: report.format === "gamecube-dol" ? 2 : null,
},
},
});
if (report.format !== "gamecube-dol") continue;
const page = await call("inspect_analysis_view", {
source,
view: {
kind: "native-load-image",
collection: "sections",
indices: [1, 0, 1],
offset: 1,
limit: 1,
fields: ["kind", "address"],
},
});
expect(page.isError, JSON.stringify(page.content)).not.toBe(true);
const section = analysisViewResultSchema.parse(
parseEvidence(page.structuredContent).normalized_result,
);
if (section.kind !== "native-load-image") throw new Error("Expected page");
expect(section.items.rows).toEqual([
{
source_index: 1,
record: { kind: "data", address: "ram:0x80004000" },
},
]);
expect(section).toMatchObject({
verification_status: "mismatch",
parent_evidence_id: parent.evidence_id,
items: {
collection: "sections",
rows: [
{
source_index: 1,
record: { kind: "data", address: "ram:0x80004000" },
},
],
},
coverage: { total: 2, next_offset: null, exhausted: true },
limitations: expect.arrayContaining(report.limitations),
});
}
}, 30_000);
+32 -2
View File
@@ -5,7 +5,14 @@ import { describe, expect, it } from "vitest";
import { z } from "zod";
import { advertisedInputExamples } from "../../../src/contracts/advertisedInputExamples.js";
import { TOOL_CONTRACTS } from "../../../src/contracts/toolContracts.js";
import {
TOOL_CONTRACTS,
toolContract,
} from "../../../src/contracts/toolContracts.js";
import {
ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES,
presentInputJsonSchema,
} from "../../../src/contracts/inputSchemaPresentation.js";
import { emptyArraySchema } from "../../../src/domain/emptyArraySchema.js";
import { processScenarioSchema } from "../../../src/domain/process/processScenario.js";
import { GENERATED_MCP_TOOL_CATALOG } from "../../fixtures/mcpToolCatalog.js";
@@ -539,8 +546,19 @@ describe("MCP JSON Schema validity", () => {
expect(recursiveReferences(tools)).toEqual([]);
expectKnownAuthorityHints(tools);
const byName = new Map(tools.map((tool) => [tool.name, tool]));
// The advertisement is fitted to the client size limit. Check the
// relaxed producer union in the complete self-contained projection.
const completeCaptureInput = presentInputJsonSchema(
toolContract("compare_web_captures").inputSchema[
"~standard"
].jsonSchema.input({
target: "draft-2020-12",
libraryOptions: { reused: "inline" },
}) as Record<string, unknown>,
() => undefined,
);
for (const side of ["before", "after"])
expect(byName.get("compare_web_captures")?.inputSchema).toHaveProperty(
expect(completeCaptureInput).toHaveProperty(
[
"properties",
side,
@@ -887,6 +905,16 @@ it("ships input schemas without local references in the generated catalog", () =
expect(localInputReferences(GENERATED_MCP_TOOL_CATALOG)).toEqual([]);
});
it("ships every input schema within the client per-tool size limit in the generated catalog", () => {
// Claude Desktop drops a tool whose compact-JSON input schema exceeds 64 KiB.
expect(
GENERATED_MCP_TOOL_CATALOG.flatMap(({ name, inputSchema }) => {
const bytes = Buffer.byteLength(JSON.stringify(inputSchema), "utf8");
return bytes > ADVERTISED_INPUT_SCHEMA_LIMIT_BYTES ? [[name, bytes]] : [];
}),
).toEqual([]);
});
it("rejects unresolved output-schema references even for uncalled tools", () => {
const errors = schemaErrors([
{
@@ -955,6 +983,8 @@ it("checks client array constraints in schema positions while preserving literal
for (const [literal, reason] of [
[{ const: "" }, "const is an empty string"],
[{ enum: ["named", ""] }, "enum contains an empty string"],
[{ type: "integer", const: 2 }, "const is a number"],
[{ enum: ["named", 1.5] }, "enum contains a number"],
] as const) {
expect(
unsupportedInputLiteralSchemaError(
@@ -26,6 +26,7 @@ type FixtureMode =
| "future_id"
| "analysis_timeout"
| "remote_error"
| "remote_startup_error"
| "shutdown_error"
| "invalid_shutdown_ack"
| "hang_after_start"
@@ -950,6 +950,71 @@ describe("GhidraClient captured source snapshot", () => {
});
});
it("scopes authenticated input-failure diagnostics to the request while retaining session output", async () => {
const client = clientFor(new FixtureLauncher("remote_error"));
const result = await client.callTool("list_procedures", {
document: null,
offset: 0,
limit: 500,
});
expect(result).toMatchObject({
ok: false,
error: {
kind: "remote",
remoteCode: "not_found",
diagnostics: {
diagnostic_scope: "request",
remote_code: "not_found",
remote_message: "Unknown Ghidra procedure name",
failure_output: {
stdout: { text: "Fixture request rejected: missing procedure\n" },
complete: true,
},
},
},
});
if (result.ok) throw new Error("Expected remote lookup failure");
expect(result.error.diagnostics).not.toHaveProperty("stdout");
expect(result.error.diagnostics).not.toHaveProperty("stderr");
expect(result.error.diagnostics).not.toHaveProperty("launch_support");
expect(client.diagnostics()).toMatchObject({
stdout: {
text: "Fixture launcher startup output\nFixture request rejected: missing procedure\n",
},
});
expect(client.isReady()).toBe(true);
await expect(
client.callTool("list_strings", { document: null }),
).resolves.toMatchObject({ ok: true });
await client.close();
expect(client.diagnostics()).toMatchObject({
stdout: {
text: expect.stringContaining("Fixture launcher startup output"),
},
});
});
it("retains launcher diagnostics when an input error rejects the startup handshake", async () => {
const client = clientFor(new FixtureLauncher("remote_startup_error"));
const result = await client.start();
expect(result).toMatchObject({
ok: false,
error: {
kind: "remote",
remoteCode: "invalid_request",
diagnostics: {
stdout: {
text: expect.stringContaining("Fixture launcher startup output"),
},
},
},
});
if (result.ok) throw new Error("Expected startup failure");
expect(result.error.diagnostics).not.toHaveProperty("diagnostic_scope");
});
describe("GhidraClient", () => {
it("completes an exact, fragmented post-analysis handshake", async () => {
const launcher = new FixtureLauncher("fragmented");
@@ -1051,28 +1116,6 @@ describe("GhidraClient", () => {
});
});
it("preserves an authenticated operation failure code and diagnostics", async () => {
const client = clientFor(new FixtureLauncher("remote_error"));
const result = await client.callTool("list_procedures", {
document: null,
offset: 0,
limit: 500,
});
expect(result).toMatchObject({
ok: false,
error: {
kind: "remote",
remoteCode: "not_found",
diagnostics: {
remote_code: "not_found",
remote_message: "Unknown Ghidra procedure name",
},
},
});
});
it.each([
["wrong_identity", "protocol"],
["malformed", "protocol"],
+32 -9
View File
@@ -62,6 +62,11 @@ const cutIdentifierAtBudget = async () => {
const writeThen = (stream, text, then) =>
stream.write(text, () => setTimeout(then, 10));
if (mode === "remote_error" || mode === "remote_startup_error")
await new Promise((resolve) =>
process.stdout.write("Fixture launcher startup output\n", resolve),
);
let endpointPublication;
if (mode === "exit") process.exit(73);
if (mode === "flood_silent") {
@@ -199,15 +204,20 @@ const handleRequest = (socket, server, request, state) => {
return;
}
if (state.mode === "remote_error" && request.method === "list_procedures") {
socket.write(
`${JSON.stringify({
id: request.id,
ok: false,
error: {
code: "not_found",
message: "Unknown Ghidra procedure name",
},
})}\n`,
writeThen(
process.stdout,
"Fixture request rejected: missing procedure\n",
() =>
socket.write(
`${JSON.stringify({
id: request.id,
ok: false,
error: {
code: "not_found",
message: "Unknown Ghidra procedure name",
},
})}\n`,
),
);
return;
}
@@ -266,6 +276,19 @@ const handleRequest = (socket, server, request, state) => {
};
const handlePing = (socket, request, state) => {
if (state.mode === "remote_startup_error") {
socket.write(
`${JSON.stringify({
id: request.id,
ok: false,
error: {
code: "invalid_request",
message: "Fixture rejected startup handshake",
},
})}\n`,
);
return;
}
state.pings.value += 1;
if (state.mode === "diagnostic_output" && state.pings.value === 1) {
const retainedTokenPrefixBytes = Math.floor(state.token.length / 2);
+7 -1
View File
@@ -40,7 +40,7 @@ export const arrayItemSchemaError = (
return undefined;
};
/** Reject Gemini-incompatible boolean and empty-string input literals. */
/** Reject Gemini-incompatible boolean, numeric and empty-string input literals. */
export const unsupportedInputLiteralSchemaError = (
schema: unknown,
path: string,
@@ -48,6 +48,7 @@ export const unsupportedInputLiteralSchemaError = (
if (!isRecord(schema)) return undefined;
if (typeof schema.const === "boolean") return `${path}.const is boolean`;
if (schema.const === "") return `${path}.const is an empty string`;
if (typeof schema.const === "number") return `${path}.const is a number`;
if (
Array.isArray(schema.enum) &&
schema.enum.some((value) => typeof value === "boolean")
@@ -55,6 +56,11 @@ export const unsupportedInputLiteralSchemaError = (
return `${path}.enum contains a boolean`;
if (Array.isArray(schema.enum) && schema.enum.includes(""))
return `${path}.enum contains an empty string`;
if (
Array.isArray(schema.enum) &&
schema.enum.some((value) => typeof value === "number")
)
return `${path}.enum contains a number`;
for (const [key, value] of Object.entries(schema)) {
const children =
schemaMaps.has(key) && isRecord(value)
+143
View File
@@ -0,0 +1,143 @@
import type {
NativeLoadImage,
NativeLoadImageObservation,
} from "../../src/domain/native/nativeLoadImage.js";
/** Authored domain observations; these projection fixtures run no provider. */
export const nativeLoadImageObservationFixture =
(): NativeLoadImageObservation => ({
executable_format: "Old-style DOS Executable (MZ)",
language_id: "x86:LE:16:Real Mode",
compiler_spec_id: "default",
image_base: "0x0",
default_address_space: "ram",
source_files: [
{
name: "fixture.exe",
size: 64,
original_sha256: "a".repeat(64),
modified_sha256: "a".repeat(64),
},
],
mappings: [
{
block: "module",
start: "ram:0x0010",
end: "ram:0x001f",
address_space: "ram",
initialized: true,
loaded: true,
overlay: false,
length: 16,
source_file_index: 0,
file_offset: 32,
sha256: "b".repeat(64),
},
],
relocations: [],
entry_points: ["ram:0x0010"],
entry_context: [],
});
/** A recorded producer verification claim, not a runtime attestation by this fixture. */
export const nativeDosLoadImageFixture = (): Extract<
NativeLoadImage,
{ format: "dos-mz" | "dos-com" }
> => ({
status: "verified",
format: "dos-mz",
target_sha256: "a".repeat(64),
load_segment: 0,
header_bytes: 32,
module_bytes: 32,
overlay_bytes: 0,
entry: { relative_segment: 0, offset: 16, linear_address: "ram:0x0010" },
observations: nativeLoadImageObservationFixture(),
checks: [
{
name: "original_file_sha256",
matched: true,
expected: "a".repeat(64),
observed: "a".repeat(64),
file_offset: null,
address: null,
},
],
limitations: [],
});
export const nativeRawLoadImageFixture = (): Extract<
NativeLoadImage,
{ format: "raw-binary" }
> => ({
status: "verified",
format: "raw-binary",
target_sha256: "c".repeat(64),
architecture: "x86_64",
language_id: "x86:LE:64:default",
compiler_spec_id: "gcc",
base_address: "ram:0x00400000",
file_bytes: 32,
entry: { address: "ram:0x00400000", file_offset: 0 },
observations: {
...nativeLoadImageObservationFixture(),
executable_format: "Raw Binary",
language_id: "x86:LE:64:default",
compiler_spec_id: "gcc",
image_base: "ram:0x00400000",
entry_points: ["ram:0x00400000"],
},
checks: [],
limitations: [],
});
export const nativeDolLoadImageFixture = (): Extract<
NativeLoadImage,
{ format: "gamecube-dol" }
> => ({
status: "mismatch",
format: "gamecube-dol",
target_sha256: "d".repeat(64),
architecture: "powerpc",
language_id: "PowerPC:BE:32:default",
compiler_spec_id: "default",
file_bytes: 288,
entry: { address: "ram:0x80003100" },
sections: [
{
kind: "text",
index: 0,
file_offset: "0x100",
address: "ram:0x80003100",
size: "0x10",
},
{
kind: "data",
index: 0,
file_offset: "0x110",
address: "ram:0x80004000",
size: "0x10",
},
],
bss: { address: "ram:0x80005000", size: "0x20" },
observations: {
...nativeLoadImageObservationFixture(),
executable_format: "Nintendo GameCube DOL",
language_id: "PowerPC:BE:32:default",
image_base: "ram:0x80003100",
entry_points: ["ram:0x80003100"],
},
checks: [
{
name: "bss",
matched: false,
expected: "0x20",
observed: "0x10",
file_offset: null,
address: "ram:0x80005000",
},
],
limitations: [
"Fixture records a BSS mismatch; this is not runtime coverage.",
],
});