test: prefer structured assertions over recovery prose (#6379)

* test: prefer structured assertions over recovery prose

Lock regex, IDA, Hopper/Kuna, MCP budget, capture-heap, and pcap
failures on error codes and reported limits instead of full recovery
sentences.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: keep decoder heap negatives off the Windows reserve

"512 MiB" also appears in the Bun Windows reserve sentence. The
negative check stays on "512 MiB (source setting)", which is the other
adapter's heap.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: keep the one-retry cap and deadline recovery tokens

The lost-connection guidance must say once only, so an unlimited
"whenever it is safe" rewrite fails. Deadline recovery still has to
name set, restart, the floor, and the maximum constant.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: keep heap, MCP, pcap, and preload recovery tokens

Codex still wanted the set/restart/floor/maximum heap guidance, MCP
default/floor/maximum bounds, Wireshark install guidance, and the
dropped-preload attribution limit as separate tokens.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: keep shared-context and transport recovery tokens

Shared-context capture must still exclude requests without a known root
or descendant frame. Shutdown must end the REA process before starting
again, and a failed HTTP listen must offer another port or stopping the
process that holds it.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: keep handoff, input-check, and budget format tokens

Skill and server guidance must still tell the user to restart the MCP
client. Process failures must still say to check the input files. Invalid
MCP phase budgets must still say positive decimal integer and ms.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: bind budget floors and active deadline sources

The phase-budget minimum is the start of "from N to M ms", so a digit
inside the maximum no longer satisfies it. Capability text must contain
the active "N ms (source ...)" pair, not the default number and a heap
source mentioned apart.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: bind heap MiB sources and pcap host tokens

A configured 256 MiB request must appear as "256 MiB (source setting)",
so a byte-sized 256 no longer passes. Unsupported pcap hosts must still
name Linux, util-linux, Windows, and owned Job Objects as separate tokens.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: bind the selected decoder heap to MiB and source

The two-adapter limitation must contain "256 MiB (source setting)".
The exhaustion message binds that capacity as "256 MiB V8 old-generation
heap (source setting)", so a byte-sized 256 no longer passes.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: bind doctor client seconds and prlimit absolute guidance

Doctor recovery must name startup_timeout_sec of the policy's seconds,
so a hardcoded 60 no longer passes while the policy stays 30. A relative
prlimit must say the value must be absolute and tell the user to check
REA_TSHARK_PRLIMIT_COMMAND.

Co-authored-by: morluto <morluto@users.noreply.github.com>

* test: assert integrity codes for rehashed evidence and coverage

Independently rehashed process Evidence and a tampered digest project
evidence_integrity_mismatch. An unsupported comparison operation stays
invalid_request. Unsupported inventory coverage status is an integrity
failure whose reason carries the coverage.status schema issue.

Co-authored-by: morluto <morluto@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-10-23 04:47:39 +08:00
1 parent 46217c69bc
commit bea6b98e84
18 files changed
+540 -282

No files matched your search

@@ -146,8 +146,8 @@ it("projects hook retention and dropped event families through the result schema
expect(result.coverage.unavailable_event_families).not.toContain(
"preload-configuration",
);
expect(result.limitations).not.toContain(
"Preload configuration was not exposed by the observed Electron window metadata. Preload paths and execution remain unavailable.",
expect(result.limitations.join("\n")).not.toContain(
"Preload configuration was not exposed",
);
});
@@ -255,7 +255,11 @@ it("does not correlate legacy dropped families and roles", () => {
{ family: "preload", count: 1 },
{ family: "renderer-ipc", count: 1 },
]);
expect(result.limitations).toContain(
"Dropped preload-family events were not attributed to preload configuration; independently aggregated event families and roles cannot establish that relationship.",
const limitations = result.limitations.join("\n");
expect(limitations).toContain("Dropped preload-family events");
expect(limitations).toContain("not attributed to preload configuration");
expect(limitations).toContain(
"independently aggregated event families and roles",
);
expect(limitations).toContain("cannot establish that relationship");
});
@@ -32,9 +32,10 @@ it("omits a request without a frame in a shared context", () => {
};
context.emit("request", unknownRequest);
expect(events.result().items).toEqual([]);
expect(events.limitations()).toContain(
"Shared-context network capture excludes requests without a known root or descendant Page frame.",
);
const limitations = events.limitations().join("\n");
expect(limitations).toContain("Shared-context network capture");
expect(limitations).toContain("excludes requests");
expect(limitations).toContain("without a known root or descendant");
});
it("detaches root and popup listeners when a scenario finishes", async () => {
+12 -4
View File
@@ -1,5 +1,9 @@
import { expect, it } from "vitest";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import {
CAPTURE_DECODER_OLD_GENERATION_SETTING,
DEFAULT_CAPTURE_DECODER_HEAP,
} from "../../config/captureDecoderHeap.js";
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
import { analysisErrorProjectionSchema } from "../../contracts/errorSchemas.js";
import { inspectWebNetworkCaptureInputSchema } from "../../domain/webNetworkCapture.js";
@@ -37,12 +41,10 @@ it.each([
code: "resource_constraint",
retryable: false,
message: expect.stringContaining(
"exhausted its 192 MiB V8 old-generation heap (source default)",
`${String(DEFAULT_CAPTURE_DECODER_HEAP.oldGenerationMiB)} MiB`,
),
remediation: {
action: expect.stringContaining(
"The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
),
action: expect.stringContaining(CAPTURE_DECODER_OLD_GENERATION_SETTING),
},
details: {
resource: "memory",
@@ -53,6 +55,12 @@ it.each([
captured_output: { stdout: "", stderr },
},
});
expect(projected.message).toContain(
`source ${DEFAULT_CAPTURE_DECODER_HEAP.source}`,
);
expect(projected.remediation.action).toContain(
`source ${DEFAULT_CAPTURE_DECODER_HEAP.source}`,
);
expect(JSON.stringify(projected)).not.toContain("rea doctor");
expect(JSON.stringify(projected)).not.toContain("no runtime override");
// Bun ignores the decoder's V8 heap flags, so no fixed heap is reported.
+21 -15
View File
@@ -52,12 +52,11 @@ it("keeps the 192 MiB decoder heap when the setting is unset", async () => {
const parsed = parseConfig({});
if (!parsed.ok) throw new Error("unset heap was rejected");
const recovery = captureDecoderHeapRecovery(DEFAULT_CAPTURE_DECODER_HEAP);
expect(recovery).toContain(CAPTURE_DECODER_OLD_GENERATION_SETTING);
expect(recovery).toContain(
"The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 192 MiB (source default; default 192 MiB, floor 8 MiB, maximum 16384 MiB).",
);
expect(recovery).toContain(
"Set REA_CAPTURE_DECODER_OLD_GENERATION_MIB in the environment that starts REA to a decimal integer from 8 to 16384, then restart REA.",
String(DEFAULT_CAPTURE_DECODER_HEAP.oldGenerationMiB),
);
expect(recovery).toContain(`source ${DEFAULT_CAPTURE_DECODER_HEAP.source}`);
expect(recovery).not.toContain("no runtime override");
const command = await new NodeCaptureAdapter("har").command(
"/owned/request.json",
@@ -98,11 +97,17 @@ it("uses a valid REA_CAPTURE_DECODER_OLD_GENERATION_MIB override", async () => {
configured(256),
),
);
const recovery = captureDecoderHeapRecovery(configured(256));
expect(projected).toMatchObject({
code: "resource_constraint",
details: { resource: "memory" },
});
expect(projected.message).toContain(
"exhausted its 256 MiB V8 old-generation heap (source setting)",
"256 MiB V8 old-generation heap (source setting)",
);
expect(projected.remediation.action).toContain(recovery);
expect(projected.remediation.action).toContain(
CAPTURE_DECODER_OLD_GENERATION_SETTING,
);
expect(projected.remediation.action).toContain("256 MiB (source setting");
expect(projected.message).not.toContain("no runtime override");
if (process.versions.bun === undefined)
expect(projected.details).toMatchObject({
@@ -158,22 +163,23 @@ it("reports the heap the selected adapter launches when decoder environment diff
if (!reported.ok)
throw new Error(projectAnalysisError(reported.error).message);
const limitations = reported.value.limitations.join("\n");
expect(limitations).toContain(
"The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 256 MiB (source setting",
);
expect(limitations).not.toContain("192 MiB (source default)");
expect(limitations).toContain(CAPTURE_DECODER_OLD_GENERATION_SETTING);
expect(limitations).toContain("256 MiB (source setting");
expect(limitations).not.toContain("source default");
expect(limitations).not.toContain("512 MiB (source setting)");
vi.spyOn(launched, "command").mockRejectedValue(heapExhaustion());
const exhausted = await decoder.inspect(input);
if (exhausted.ok) throw new Error("Heap exhaustion must fail inspection");
const projected = projectAnalysisError(exhausted.error);
expect(projected.code).toBe("resource_constraint");
expect(projected.message).toContain(
"exhausted its 256 MiB V8 old-generation heap (source setting)",
"256 MiB V8 old-generation heap (source setting)",
);
expect(projected.remediation.action).toContain(
"The effective REA_CAPTURE_DECODER_OLD_GENERATION_MIB value is 256 MiB (source setting",
CAPTURE_DECODER_OLD_GENERATION_SETTING,
);
expect(projected.remediation.action).toContain("256 MiB (source setting");
expect(projected.message).not.toContain("192 MiB");
expect(projected.remediation.action).not.toContain(
"512 MiB (source setting)",
@@ -228,8 +234,8 @@ it.each([
});
if (startup.ok) throw new Error(`Startup accepted ${JSON.stringify(value)}`);
expect(startup.error.settings).toContainEqual({
setting: "REA_CAPTURE_DECODER_OLD_GENERATION_MIB",
constraint: expect.stringContaining("from 8 to 16384 MiB"),
setting: CAPTURE_DECODER_OLD_GENERATION_SETTING,
constraint: captureDecoderHeapConstraint,
});
const config = parseConfig({
REA_CAPTURE_DECODER_OLD_GENERATION_MIB: value,
+10 -13
View File
@@ -5,6 +5,7 @@ import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { parseEnvironment } from "./environment.js";
import {
DEFAULT_MCP_STARTUP_BUDGETS,
MCP_PHASE_BUDGET_FLOOR_MS,
MCP_PHASE_BUDGET_MAXIMUM_MS,
parseMcpStartupBudgets,
resolveMcpStartupBudgets,
@@ -76,21 +77,17 @@ it.each([
if (parsed.ok)
throw new Error(`Accepted ${setting}=${JSON.stringify(value)}`);
expect(parsed.error).toBeInstanceOf(ConfigurationError);
expect(parsed.error.settings).toEqual([
{
setting,
constraint: expect.stringContaining(
`${setting} must be a positive decimal integer from 1 to 2147483647 ms`,
),
},
]);
expect(parsed.error.message).toContain(setting);
const constraint = parsed.error.settings[0]?.constraint ?? "";
expect(parsed.error.settings).toEqual([{ setting, constraint }]);
expect(constraint).toContain(setting);
expect(constraint).toContain("positive decimal integer");
expect(constraint).toContain(
`from ${String(MCP_PHASE_BUDGET_FLOOR_MS)} to ${String(MCP_PHASE_BUDGET_MAXIMUM_MS)} ms`,
);
expect(parsed.error.message).toBe(constraint);
const startup = parseEnvironment({ [setting]: value });
if (startup.ok) throw new Error(`Startup accepted ${setting}`);
expect(startup.error.settings).toContainEqual({
setting,
constraint: expect.stringContaining("from 1 to 2147483647 ms"),
});
expect(startup.error.settings).toContainEqual({ setting, constraint });
const config = parseConfig({ [setting]: value });
if (config.ok) throw new Error(`parseConfig accepted ${setting}`);
expect(projectAnalysisError(config.error)).toMatchObject({
+48 -17
View File
@@ -20,6 +20,15 @@ import {
const configured = (deadlineMs: number) =>
({ deadlineMs, source: "setting" }) as const;
const expectDeadlineRecoveryTokens = (text: string | undefined) => {
expect(text).toContain(`Set ${REGEX_MATCHING_DEADLINE_SETTING}`);
expect(text).toContain("then restart REA");
expect(text).toContain("floor 1 ms");
expect(text).toContain(
`maximum ${String(MAXIMUM_REGEX_MATCHING_DEADLINE_MS)} ms`,
);
};
it("keeps the 5000 ms deadline when the setting is unset", () => {
expect(parseRegexMatchingDeadline(undefined)).toEqual({
ok: true,
@@ -33,15 +42,33 @@ it("keeps the 5000 ms deadline when the setting is unset", () => {
const recovery = regexMatchingDeadlineRecovery(
DEFAULT_REGEX_MATCHING_DEADLINE,
);
expect(recovery).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(recovery).toContain(
"The effective REA_REGEX_MATCHING_DEADLINE_MS value is 5000 ms (source default; default 5000 ms, floor 1 ms, maximum 2147483647 ms).",
String(DEFAULT_REGEX_MATCHING_DEADLINE.deadlineMs),
);
expect(recovery).toContain(
`source ${DEFAULT_REGEX_MATCHING_DEADLINE.source}`,
);
expectDeadlineRecoveryTokens(recovery);
const projected = projectAnalysisError(
regexMatchingDeadlineError("search_procedures"),
);
expect(projected.code).toBe("resource_constraint");
expect(projected.message).toContain(recovery);
expect(projected.remediation.action).toContain(recovery);
expect(projected).toMatchObject({
code: "resource_constraint",
details: {
resource: "cpu",
reported_limits: {
matching_deadline_ms: DEFAULT_REGEX_MATCHING_DEADLINE.deadlineMs,
matching_deadline_source: DEFAULT_REGEX_MATCHING_DEADLINE.source,
},
},
});
expect(projected.message).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(projected.remediation.action).toContain(
REGEX_MATCHING_DEADLINE_SETTING,
);
expectDeadlineRecoveryTokens(projected.message);
expectDeadlineRecoveryTokens(projected.remediation.action);
expect(projected.message).not.toContain("no runtime override");
});
@@ -66,10 +93,11 @@ it("uses a valid REA_REGEX_MATCHING_DEADLINE_MS override", () => {
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" ");
expect(hopperText).toContain("10000 ms matching deadline (source setting)");
expect(hopperText).toContain("REA_REGEX_MATCHING_DEADLINE_MS");
expect(hopperText).toContain("default 5000 ms");
expect(hopperText).toContain("maximum 2147483647 ms");
expect(hopperText).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(hopperText).toContain(
`${String(parsed.value.regexMatchingDeadline.deadlineMs)} ms (source setting`,
);
expect(hopperText).not.toContain("1 ms matching deadline");
expect(hopperText).not.toContain("no runtime override");
vi.stubEnv(REGEX_MATCHING_DEADLINE_SETTING, "1");
@@ -79,19 +107,22 @@ it("uses a valid REA_REGEX_MATCHING_DEADLINE_MS override", () => {
.capabilities()
.find((capability) => capability.operation === "search_strings")
?.limitations.join(" ");
expect(kunaText).toContain("10000 ms matching deadline (source setting)");
expect(kunaText).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(kunaText).toContain("10000 ms (source setting");
expect(kunaText).not.toContain("1 ms matching deadline");
} finally {
vi.unstubAllEnvs();
}
const ambient = new KunaProvider({});
expect(
ambient
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" "),
).toContain("5000 ms matching deadline (source default)");
const ambientText = ambient
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" ");
expect(ambientText).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(ambientText).toContain(
`${String(DEFAULT_REGEX_MATCHING_DEADLINE.deadlineMs)} ms (source ${DEFAULT_REGEX_MATCHING_DEADLINE.source}`,
);
});
it.each([
@@ -120,8 +151,8 @@ it.each([
const startup = parseEnvironment({ REA_REGEX_MATCHING_DEADLINE_MS: value });
if (startup.ok) throw new Error(`Startup accepted ${JSON.stringify(value)}`);
expect(startup.error.settings).toContainEqual({
setting: "REA_REGEX_MATCHING_DEADLINE_MS",
constraint: expect.stringContaining("from 1 to 2147483647 ms"),
setting: REGEX_MATCHING_DEADLINE_SETTING,
constraint: regexMatchingDeadlineConstraint,
});
const config = parseConfig({ REA_REGEX_MATCHING_DEADLINE_MS: value });
if (config.ok)
+44 -21
View File
@@ -21,6 +21,17 @@ import {
const configured = (heapMiB: number) =>
({ heapMiB, source: "setting" }) as const;
const expectHeapRecoveryTokens = (text: string | undefined) => {
expect(text).toContain(`Set ${REGEX_WORKER_HEAP_REQUEST_SETTING}`);
expect(text).toContain("then restart REA");
expect(text).toContain(
`floor ${String(MINIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB`,
);
expect(text).toContain(
`maximum ${String(MAXIMUM_REGEX_WORKER_HEAP_REQUEST_MIB)} MiB`,
);
};
it("keeps the 128 MiB request when the setting is unset", () => {
expect(parseRegexWorkerHeapRequest(undefined)).toEqual({
ok: true,
@@ -34,17 +45,29 @@ it("keeps the 128 MiB request when the setting is unset", () => {
const recovery = regexWorkerHeapRequestRecovery(
DEFAULT_REGEX_WORKER_HEAP_REQUEST,
);
expect(recovery).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(recovery).toContain(String(DEFAULT_REGEX_WORKER_HEAP_REQUEST.heapMiB));
expect(recovery).toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 128 MiB (source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB).",
);
expect(recovery).toContain(
"Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA",
`source ${DEFAULT_REGEX_WORKER_HEAP_REQUEST.source}`,
);
expectHeapRecoveryTokens(recovery);
const projected = projectAnalysisError(
regexWorkerHeapError("search_procedures", new Error("heap")),
);
expect(projected.code).toBe("resource_constraint");
expect(projected.remediation.action).toContain(recovery);
expect(projected).toMatchObject({
code: "resource_constraint",
details: {
resource: "memory",
reported_limits: {
requested_worker_heap_mib: DEFAULT_REGEX_WORKER_HEAP_REQUEST.heapMiB,
requested_worker_heap_source: DEFAULT_REGEX_WORKER_HEAP_REQUEST.source,
},
},
});
expect(projected.remediation.action).toContain(
REGEX_WORKER_HEAP_REQUEST_SETTING,
);
expectHeapRecoveryTokens(projected.remediation.action);
expect(projected.remediation.action).not.toContain("no runtime override");
});
@@ -76,12 +99,9 @@ it("uses a valid REA_REGEX_WORKER_HEAP_REQUEST_MIB override", () => {
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" ");
expect(hopperText).toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).",
);
expect(hopperText).toContain(
"Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA",
);
expect(hopperText).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(hopperText).toContain("256 MiB (source setting");
expect(hopperText).not.toMatch(/\b8 MiB\b/u);
expect(hopperText).not.toContain("no runtime override");
vi.stubEnv(REGEX_WORKER_HEAP_REQUEST_SETTING, "8");
@@ -93,24 +113,27 @@ it("uses a valid REA_REGEX_WORKER_HEAP_REQUEST_MIB override", () => {
.capabilities()
.find((capability) => capability.operation === "search_strings")
?.limitations.join(" ");
expect(kunaText).toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).",
);
expect(kunaText).not.toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 8 MiB",
);
expect(kunaText).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(kunaText).toContain("256 MiB (source setting");
expect(kunaText).not.toMatch(/\b8 MiB\b/u);
} finally {
vi.unstubAllEnvs();
}
const ambient = new KunaProvider({});
expect(
ambient
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" "),
).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(
ambient
.capabilities()
.find((capability) => capability.operation === "search_procedures")
?.limitations.join(" "),
).toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 128 MiB (source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB).",
`${String(DEFAULT_REGEX_WORKER_HEAP_REQUEST.heapMiB)} MiB (source ${DEFAULT_REGEX_WORKER_HEAP_REQUEST.source}`,
);
});
@@ -143,8 +166,8 @@ it.each([
});
if (startup.ok) throw new Error(`Startup accepted ${JSON.stringify(value)}`);
expect(startup.error.settings).toContainEqual({
setting: "REA_REGEX_WORKER_HEAP_REQUEST_MIB",
constraint: expect.stringContaining("from 7 to 16384 MiB"),
setting: REGEX_WORKER_HEAP_REQUEST_SETTING,
constraint: regexWorkerHeapRequestConstraint,
});
const config = parseConfig({ REA_REGEX_WORKER_HEAP_REQUEST_MIB: value });
if (config.ok)
+9 -1
View File
@@ -7,13 +7,16 @@ import {
RecordingIdaMcp,
} from "../../tests/fixtures/idaMcp.js";
import { IdaSessionClient } from "./IdaSessionClient.js";
import {
DEFAULT_REGEX_MATCHING_DEADLINE,
REGEX_MATCHING_DEADLINE_SETTING,
} from "../config/regexMatchingDeadline.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import {
AnalysisProtocolError,
AnalysisResourceConstraintError,
} from "../domain/analysisErrorCore.js";
import { DEFAULT_REGEX_MATCHING_DEADLINE } from "../config/regexMatchingDeadline.js";
import { ProviderAdapterError } from "../domain/providerAdapterError.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import { WindowsPrivateRuntime } from "../windows/WindowsPrivateRuntime.js";
@@ -590,6 +593,7 @@ describe("attached IDA regex supervision", () => {
matching_deadline_source: DEFAULT_REGEX_MATCHING_DEADLINE.source,
},
});
expect(result.error.message).toContain(REGEX_MATCHING_DEADLINE_SETTING);
const recovered = await client.execute("search_strings", {
pattern: "!$",
mode: "regex",
@@ -597,6 +601,10 @@ describe("attached IDA regex supervision", () => {
expect(recovered.ok && recovered.value.result).toEqual([
expect.objectContaining({ address: "0x3000", value }),
]);
if (recovered.ok)
expect(recovered.value.limitations.join("\n")).toContain(
REGEX_MATCHING_DEADLINE_SETTING,
);
} finally {
await client.close();
}
+38 -30
View File
@@ -32,38 +32,44 @@ function prose(text: string): string {
}
it("uses one retry then a user handoff for transient MCP failures", () => {
expect(MCP_CONNECTION_LOST).toBe(
"Retry the tool call once only when it is safe to repeat or known not to have started. If it may already have changed the target, or it fails again, tell the user REA lost its MCP client connection and ask them to restart the MCP client.",
);
expect(MCP_CONNECTION_LOST).toMatch(/^Retry /u);
// "once only" is the single-attempt cap. "Retry whenever it is safe to
// repeat" does not contain it.
expect(MCP_CONNECTION_LOST).toContain("once only");
expect(MCP_CONNECTION_LOST).toContain("safe to repeat");
expect(MCP_CONNECTION_LOST).toContain("tell the user");
expect(MCP_CONNECTION_LOST).toContain("restart the MCP client");
expect(MCP_CONNECTION_LOST).not.toContain("rea doctor");
expect(MCP_CONNECTION_START_FAILED).toBe(
"Retry once. If it fails again, tell the user REA could not start its MCP connection and ask them to restart the MCP client, then run `rea doctor`.",
);
expect(MCP_CONNECTION_START_FAILED).toContain("Retry once.");
expect(SERVER_START_FAILED).toBe(
"Do not repeat this startup. Tell the user REA could not start and ask them to restart the MCP client, then run `rea doctor`.",
);
expect(MCP_CONNECTION_START_FAILED).toMatch(/^Retry once\./u);
expect(SERVER_START_FAILED).toMatch(/^Do not repeat /u);
expect(SERVER_START_FAILED).not.toMatch(/\bretry\b/iu);
for (const message of [MCP_CONNECTION_START_FAILED, SERVER_START_FAILED]) {
expect(message).toMatch(/tell the user/iu);
expect(message).toContain("restart the MCP client");
expect(message).toContain("`rea doctor`");
}
expect(PROCESS_UNEXPECTED_FAILURE_REMEDIATION).toBe(
"Do not repeat this command. Tell the user the process command could not complete and ask them to check the input files, then run `rea doctor`.",
expect(PROCESS_UNEXPECTED_FAILURE_REMEDIATION).toMatch(/^Do not repeat /u);
expect(PROCESS_UNEXPECTED_FAILURE_REMEDIATION).toContain(
"check the input files",
);
expect(PROCESS_UNEXPECTED_FAILURE_REMEDIATION).not.toMatch(/\bretry\b/iu);
for (const message of [
MCP_CONNECTION_START_FAILED,
SERVER_START_FAILED,
PROCESS_UNEXPECTED_FAILURE_REMEDIATION,
]) {
expect(message).toMatch(/tell the user/iu);
expect(message).toContain("`rea doctor`");
}
for (const message of [MCP_CONNECTION_START_FAILED, SERVER_START_FAILED])
expect(message).toContain("restart the MCP client");
});
it("keeps non-retryable transport failures on their named fix", () => {
expect(MCP_SHUTDOWN_FAILED).toBe(
"REA could not close cleanly. End the REA process before starting it again.",
);
expect(MCP_HTTP_LISTEN_FAILED).toBe(
"REA could not listen for MCP HTTP connections on the selected host and port. Choose another --port or stop the process using it.",
);
expect(MCP_SHUTDOWN_FAILED).toContain("close cleanly");
expect(MCP_SHUTDOWN_FAILED).toContain("End the REA process");
expect(MCP_SHUTDOWN_FAILED).toContain("before starting it again");
expect(MCP_HTTP_LISTEN_FAILED).toContain("Choose another --port");
expect(MCP_HTTP_LISTEN_FAILED).toContain("stop the process using it");
expect(MCP_SHUTDOWN_FAILED).not.toContain("Retry");
expect(MCP_SHUTDOWN_FAILED).not.toContain("tell the user");
expect(MCP_HTTP_LISTEN_FAILED).not.toContain("tell the user");
expect(MCP_HTTP_LISTEN_FAILED).not.toContain("Retry");
});
it("keeps ceiling dial recoveries and scrubs non-adjustable timeout teaching", async () => {
@@ -113,10 +119,11 @@ it("keeps ceiling dial recoveries and scrubs non-adjustable timeout teaching", a
"utf8",
);
const skillProse = prose(skill);
expect(skillProse).toContain(
"retry that call once only when it is safe to repeat or known not to have started",
);
expect(skillProse).toContain("tell the user REA's connection failed");
expect(skillProse).toContain("once only");
expect(skillProse).toContain("safe to repeat");
expect(skillProse).toContain("tell the user");
expect(skillProse).toContain("restart the MCP client");
expect(skillProse).toContain("connection failed");
expect(skillProse).toContain("Do not invent a setting");
expect(skillProse).toContain("explicitly recommends that change");
expect(skillProse).toContain("not a recommendation to change it");
@@ -124,9 +131,10 @@ it("keeps ceiling dial recoveries and scrubs non-adjustable timeout teaching", a
const instructions = prose(
await readFile(join(root, "src/server/createServer.ts"), "utf8"),
);
expect(instructions).toContain(
"retry that call once only when it is safe to repeat or known not to have started",
);
expect(instructions).toContain("once only");
expect(instructions).toContain("safe to repeat");
expect(instructions).toContain("tell the user");
expect(instructions).toContain("restart the MCP client");
expect(instructions).toContain("explicitly recommends that change");
expect(instructions).toContain("not a recommendation to change it");
expect(instructions).not.toContain("names a request field");
+14 -20
View File
@@ -1,6 +1,10 @@
import { expect, it } from "vitest";
import { DEFAULT_REGEX_MATCHING_DEADLINE } from "../config/regexMatchingDeadline.js";
import {
DEFAULT_REGEX_MATCHING_DEADLINE,
REGEX_MATCHING_DEADLINE_SETTING,
} from "../config/regexMatchingDeadline.js";
import { REGEX_WORKER_HEAP_REQUEST_SETTING } from "../config/regexWorkerHeapRequest.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
import {
execArgvFromCommandLine,
@@ -451,20 +455,12 @@ it("describes the effective heap and the configurable matching deadline", () =>
{ kind: "unknown" } as const,
]) {
const limitation = regexMatchingLimitation(heap);
expect(limitation).toContain("5000 ms matching deadline (source default)");
expect(limitation).toContain("REA_REGEX_MATCHING_DEADLINE_MS");
expect(limitation).toContain("floor 1 ms");
expect(limitation).toContain("maximum 2147483647 ms");
expect(limitation).toContain("REA_REGEX_WORKER_HEAP_REQUEST_MIB");
expect(limitation).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(limitation).toContain(
"source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB",
);
expect(limitation).toContain(
"Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA",
`${String(DEFAULT_REGEX_MATCHING_DEADLINE.deadlineMs)} ms (source ${DEFAULT_REGEX_MATCHING_DEADLINE.source}`,
);
expect(limitation).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(limitation).not.toContain("no runtime override");
expect(limitation).not.toContain("either cap");
expect(limitation).not.toContain("no runtime override for the heap");
const projected = projectAnalysisError(
regexWorkerHeapError("search_procedures", new Error("heap"), heap),
);
@@ -521,18 +517,16 @@ it("keeps a selected heap request unless a positive Node flag overrides it", ()
request,
);
expect(selected).toContain("effective 256 MiB");
expect(selected).toContain(
"The effective REA_REGEX_WORKER_HEAP_REQUEST_MIB value is 256 MiB (source setting; default 128 MiB, floor 7 MiB, maximum 16384 MiB).",
);
expect(selected).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(selected).toContain("256 MiB (source setting");
const overridden = regexMatchingLimitation(
effective(64, true),
DEFAULT_REGEX_MATCHING_DEADLINE,
request,
);
expect(overridden).toContain("effective 64 MiB");
expect(overridden).toContain("source setting");
expect(overridden).toContain("floor 7 MiB");
expect(overridden).toContain("maximum 16384 MiB");
expect(overridden).toContain(REGEX_WORKER_HEAP_REQUEST_SETTING);
expect(overridden).toContain("256 MiB (source setting");
const projected = projectAnalysisError(
regexWorkerHeapError(
"search_procedures",
@@ -548,9 +542,9 @@ it("keeps a selected heap request unless a positive Node flag overrides it", ()
requested_worker_heap_source: "setting",
},
});
expect(projected.remediation.action).toContain("source setting");
expect(projected.remediation.action).toContain(
"Set REA_REGEX_WORKER_HEAP_REQUEST_MIB in the environment that starts REA",
REGEX_WORKER_HEAP_REQUEST_SETTING,
);
expect(projected.remediation.action).toContain("256 MiB (source setting");
expect(projected.remediation.action).not.toContain("no runtime override");
});
@@ -90,16 +90,36 @@ cliTest(
],
});
expect(result.exitCode).toBe(1);
expect(result.json).toMatchObject({
code: "invalid_request",
details: {
operation: "project_android_application_graph",
issues: expect.arrayContaining([
expect.objectContaining({
message: expect.stringContaining("coverage.status"),
}),
]),
const projected = result.json;
expect(projected).toMatchObject({
code: "evidence_integrity_mismatch",
category: "integrity_mismatch",
});
const reason = inventoryIntegrityReason(projected);
expect(JSON.parse(reason)).toMatchObject([
{
code: "invalid_value",
values: ["complete", "unknown"],
path: ["coverage", "status"],
},
]);
expect(projected).toMatchObject({
message: reason,
details: { reason },
});
},
);
const inventoryIntegrityReason = (projected: unknown): string => {
if (
projected === null ||
typeof projected !== "object" ||
!("details" in projected) ||
projected.details === null ||
typeof projected.details !== "object" ||
!("reason" in projected.details) ||
typeof projected.details.reason !== "string"
)
throw new Error("Inventory rejection omitted an integrity reason");
return projected.details.reason;
};
+26 -28
View File
@@ -19,8 +19,24 @@ import {
createCatalogIdentity,
} from "../../../src/catalogIdentity.js";
import { compactPresentationForMcpInputSchemaProfile } from "../../../src/config/mcpInputSchemaProfile.js";
import {
MCP_PHASE_BUDGET_FLOOR_MS,
MCP_PHASE_BUDGET_MAXIMUM_MS,
} from "../../../src/config/mcpStartupBudgets.js";
import { MCP_STARTUP_POLICY } from "../../../src/mcpStartupPolicy.js";
const expectDoctorDeadline = (
detail: string | undefined,
milliseconds: number,
) => {
expect(detail).toContain("REA_MCP_DOCTOR_DEADLINE_MS");
expect(detail).toContain(`${String(milliseconds)} ms (source setting`);
expect(detail).toContain(
`startup_timeout_sec of ${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)} seconds`,
);
expect(detail).not.toContain("no runtime override");
};
afterEach(() => vi.useRealTimers());
const assertCliCatalogIdentity = async (
@@ -410,16 +426,8 @@ describe("production MCP doctor child lifecycle", () => {
const result = await vi.waitFor(() => pending, { timeout: 10_000 });
vi.useRealTimers();
expect(result.healthy).toBe(false);
expect(result.checks[0]?.detail).toContain(
"The effective REA_MCP_DOCTOR_DEADLINE_MS value for the MCP doctor startup deadline is 60000 ms (source setting; default 30000 ms, floor 1 ms, maximum 2147483647 ms).",
);
expect(result.checks[0]?.detail).toContain(
"Set REA_MCP_DOCTOR_DEADLINE_MS in the environment that starts REA",
);
expect(result.checks[0]?.detail).not.toContain("no runtime override");
expect(result.checks[0]?.detail).toContain(
"Codex startup_timeout_sec of 30 seconds is client guidance",
);
expectDoctorDeadline(result.checks[0]?.detail, 60_000);
expect(result.checks[0]?.detail).toContain("client guidance");
expect(result.checks[0]?.detail).not.toContain("REA_MCP_STARTUP");
await expect(waitForExit(pid)).resolves.toBeUndefined();
}, 10_000);
@@ -455,17 +463,9 @@ describe("production MCP doctor request-flow deadline", () => {
(check) => check.name === "request-flow",
);
expect(requestFlow?.ok).toBe(false);
expect(requestFlow?.detail).toContain(
"The effective REA_MCP_DOCTOR_DEADLINE_MS value for the MCP doctor startup deadline is 60000 ms (source setting; default 30000 ms, floor 1 ms, maximum 2147483647 ms).",
);
expect(requestFlow?.detail).toContain(
"Set REA_MCP_DOCTOR_DEADLINE_MS in the environment that starts REA",
);
expect(requestFlow?.detail).not.toContain("no runtime override");
expect(requestFlow?.detail).toContain(
"Codex startup_timeout_sec of 30 seconds is client guidance",
);
expect(requestFlow?.detail).not.toContain("is 30000 ms");
expectDoctorDeadline(requestFlow?.detail, 60_000);
expect(requestFlow?.detail).toContain("client guidance");
expect(requestFlow?.detail).not.toContain("source default");
expect(requestFlow?.detail).not.toContain("REA_MCP_STARTUP");
}, 15_000);
@@ -526,10 +526,7 @@ describe("production MCP doctor request-flow deadline", () => {
const result = await vi.waitFor(() => pending, { timeout: 10_000 });
vi.useRealTimers();
expect(result.healthy).toBe(false);
expect(result.checks[0]?.detail).toContain(
"The effective REA_MCP_DOCTOR_DEADLINE_MS value for the MCP doctor startup deadline is 2000 ms (source setting; default 30000 ms, floor 1 ms, maximum 2147483647 ms).",
);
expect(result.checks[0]?.detail).not.toContain("no runtime override");
expectDoctorDeadline(result.checks[0]?.detail, 2_000);
await expect(waitForExit(pid)).resolves.toBeUndefined();
}, 10_000);
@@ -550,10 +547,11 @@ describe("production MCP doctor request-flow deadline", () => {
"REA_MCP_INITIALIZE_BUDGET_MS",
"REA_MCP_DOCTOR_DEADLINE_MS",
]);
expect(result.checks[0]?.detail).toContain(
"REA_MCP_INITIALIZE_BUDGET_MS must be a positive decimal integer from 1 to 2147483647 ms",
);
const budgetRange = `from ${String(MCP_PHASE_BUDGET_FLOOR_MS)} to ${String(MCP_PHASE_BUDGET_MAXIMUM_MS)} ms`;
expect(result.checks[0]?.detail).toContain("REA_MCP_INITIALIZE_BUDGET_MS");
expect(result.checks[0]?.detail).toContain(budgetRange);
expect(result.checks[1]?.detail).toContain("REA_MCP_DOCTOR_DEADLINE_MS");
expect(result.checks[1]?.detail).toContain(budgetRange);
expect(result.checks[0]?.detail).not.toContain("no runtime override");
});
});
+76 -56
View File
@@ -14,6 +14,8 @@ import {
} from "../../../scripts/lib/mcp-startup-probe.mjs";
import {
DEFAULT_MCP_STARTUP_BUDGETS,
MCP_PHASE_BUDGET_FLOOR_MS,
MCP_PHASE_BUDGET_MAXIMUM_MS,
mcpPhaseBudgetRecovery,
resolveMcpStartupBudgets,
type McpPhaseBudget,
@@ -23,6 +25,38 @@ import {
MCP_STARTUP_POLICY,
} from "../../../src/mcpStartupPolicy.js";
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const expectPhaseText = (text: string, budget: McpPhaseBudget) => {
expect(budget.floorMilliseconds).toBe(MCP_PHASE_BUDGET_FLOOR_MS);
expect(budget.maximumMilliseconds).toBe(MCP_PHASE_BUDGET_MAXIMUM_MS);
expect(text).toContain(budget.setting);
expect(text).toContain(
`${String(budget.milliseconds)} ms (source ${budget.source}`,
);
expect(text).toContain(`default ${String(budget.defaultMilliseconds)} ms`);
expect(text).toContain(`floor ${String(MCP_PHASE_BUDGET_FLOOR_MS)} ms`);
expect(text).toContain(`maximum ${String(MCP_PHASE_BUDGET_MAXIMUM_MS)} ms`);
expect(text).not.toContain("no runtime override");
};
const expectThrownBudget = async (
pending: Promise<unknown>,
budget: McpPhaseBudget,
): Promise<string> => {
let caught: unknown;
try {
await pending;
} catch (cause) {
caught = cause;
}
expect(caught).toBeInstanceOf(Error);
const message = errorMessage(caught);
expectPhaseText(message, budget);
return message;
};
describe("MCP startup policy", () => {
it("names phase budget settings without turning Codex's 30s into a REA knob", () => {
const budgets = resolveMcpStartupBudgets({});
@@ -33,31 +67,22 @@ describe("MCP startup policy", () => {
expect(budgets.firstCatalog.milliseconds).toBe(20_000);
expect(budgets.doctor.milliseconds).toBe(30_000);
const initialize = mcpPhaseBudgetRecovery(budgets.initialize);
expect(initialize).toBe(
"The effective REA_MCP_INITIALIZE_BUDGET_MS value for the MCP initialize phase budget is 15000 ms (source default; default 15000 ms, floor 1 ms, maximum 2147483647 ms). Set REA_MCP_INITIALIZE_BUDGET_MS in the environment that starts REA to a positive decimal integer from 1 to 2147483647, then restart REA.",
);
expect(initialize).toBe(mcpStartupPhaseBudgetRecovery(budgets.initialize));
expect(initialize).not.toContain("no runtime override");
expect(mcpStartupPhaseBudgetRecovery(budgets.toolsList)).toContain(
"The effective REA_MCP_TOOLS_LIST_BUDGET_MS value for the MCP tools/list phase budget is 5000 ms (source default; default 5000 ms, floor 1 ms, maximum 2147483647 ms).",
expectPhaseText(initialize, budgets.initialize);
expectPhaseText(
mcpStartupPhaseBudgetRecovery(budgets.toolsList),
budgets.toolsList,
);
expect(mcpStartupPhaseBudgetRecovery(budgets.firstCatalog)).toContain(
"REA_MCP_FIRST_CATALOG_BUDGET_MS",
);
expect(mcpStartupPhaseBudgetRecovery(budgets.firstCatalog)).toContain(
"20000 ms (source default",
expectPhaseText(
mcpStartupPhaseBudgetRecovery(budgets.firstCatalog),
budgets.firstCatalog,
);
const doctor = mcpDoctorDeadlineRecovery(budgets.doctor);
expectPhaseText(doctor, budgets.doctor);
expect(doctor).toContain(
"The effective REA_MCP_DOCTOR_DEADLINE_MS value for the MCP doctor startup deadline is 30000 ms (source default; default 30000 ms, floor 1 ms, maximum 2147483647 ms).",
);
expect(doctor).toContain(
"Set REA_MCP_DOCTOR_DEADLINE_MS in the environment that starts REA",
);
expect(doctor).not.toContain("no runtime override");
expect(doctor).toContain(
"Codex startup_timeout_sec of 30 seconds is client guidance",
`startup_timeout_sec of ${String(MCP_STARTUP_POLICY.codexStartupTimeoutSeconds)} seconds`,
);
expect(doctor).toContain("client guidance");
expect(doctor).toContain("does not enforce");
expect(doctor).not.toContain("REA_MCP_STARTUP");
const selected = resolveMcpStartupBudgets({
@@ -136,27 +161,25 @@ const probePolicy = (
};
};
const errorMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
describe("packaged MCP startup probe failures", () => {
it("names the initialize budget when that phase times out", async () => {
const budgetMs = 200;
await expect(
const policy = probePolicy({ initializeBudgetMs: budgetMs });
await expectThrownBudget(
measureMcpStartup({
command: process.execPath,
args: [probeChild],
environment: { ...process.env, REA_MCP_PROBE_MODE: "hang" },
policy: probePolicy({ initializeBudgetMs: budgetMs }),
policy,
}),
).rejects.toThrow(
`The effective REA_MCP_INITIALIZE_BUDGET_MS value for the MCP initialize phase budget is ${String(budgetMs)} ms (source setting; default 15000 ms, floor 1 ms, maximum 2147483647 ms).`,
policy.initialize,
);
});
it("names the tools/list budget when that phase times out", async () => {
const budgetMs = 200;
await expect(
const policy = probePolicy({ toolsListBudgetMs: budgetMs });
await expectThrownBudget(
measureMcpStartup({
command: process.execPath,
args: [probeChild],
@@ -164,24 +187,23 @@ describe("packaged MCP startup probe failures", () => {
...process.env,
REA_MCP_PROBE_MODE: "initialize-then-hang",
},
policy: probePolicy({ toolsListBudgetMs: budgetMs }),
policy,
}),
).rejects.toThrow(
`The effective REA_MCP_TOOLS_LIST_BUDGET_MS value for the MCP tools/list phase budget is ${String(budgetMs)} ms (source setting; default 5000 ms, floor 1 ms, maximum 2147483647 ms).`,
policy.toolsList,
);
});
it("names the first-catalog budget when startup exceeds it", async () => {
const budgetMs = 1;
await expect(
const policy = probePolicy({ firstCatalogBudgetMs: budgetMs });
await expectThrownBudget(
measureMcpStartup({
command: process.execPath,
args: [probeChild],
environment: { ...process.env, REA_MCP_PROBE_MODE: "serve" },
policy: probePolicy({ firstCatalogBudgetMs: budgetMs }),
policy,
}),
).rejects.toThrow(
`The effective REA_MCP_FIRST_CATALOG_BUDGET_MS value for the MCP first-catalog phase budget is ${String(budgetMs)} ms (source setting; default 20000 ms, floor 1 ms, maximum 2147483647 ms).`,
policy.firstCatalog,
);
});
@@ -205,22 +227,23 @@ describe("packaged MCP startup probe failures", () => {
it("names the doctor deadline when module profiling cannot initialize", async () => {
const budgetMs = 200;
await expect(
const policy = probePolicy({ doctorDeadlineMs: budgetMs });
await expectThrownBudget(
profileMcpModuleLoading({
command: process.execPath,
args: [probeChild],
environment: { ...process.env, REA_MCP_PROBE_MODE: "hang" },
policy: probePolicy({ doctorDeadlineMs: budgetMs }),
policy,
packageName: "rea-agents",
}),
).rejects.toThrow(
`The effective REA_MCP_DOCTOR_DEADLINE_MS value for the MCP doctor startup deadline is ${String(budgetMs)} ms (source setting; default 30000 ms, floor 1 ms, maximum 2147483647 ms).`,
policy.doctor,
);
});
it("names the tools/list budget when module profiling passes initialize", async () => {
const budgetMs = 200;
await expect(
const policy = probePolicy({ toolsListBudgetMs: budgetMs });
await expectThrownBudget(
profileMcpModuleLoading({
command: process.execPath,
args: [probeChild],
@@ -228,11 +251,10 @@ describe("packaged MCP startup probe failures", () => {
...process.env,
REA_MCP_PROBE_MODE: "initialize-then-hang",
},
policy: probePolicy({ toolsListBudgetMs: budgetMs }),
policy,
packageName: "rea-agents",
}),
).rejects.toThrow(
`The effective REA_MCP_TOOLS_LIST_BUDGET_MS value for the MCP tools/list phase budget is ${String(budgetMs)} ms (source setting; default 5000 ms, floor 1 ms, maximum 2147483647 ms).`,
policy.toolsList,
);
});
@@ -260,6 +282,10 @@ describe("packaged MCP first-catalog budget while a phase is pending", () => {
it("stops a pending initialize at the shorter first-catalog budget", async () => {
const firstCatalogMs = 250;
const initializeMs = 8_000;
const policy = probePolicy({
initializeBudgetMs: initializeMs,
firstCatalogBudgetMs: firstCatalogMs,
});
const started = performance.now();
let caught: unknown;
try {
@@ -267,19 +293,14 @@ describe("packaged MCP first-catalog budget while a phase is pending", () => {
command: process.execPath,
args: [probeChild],
environment: { ...process.env, REA_MCP_PROBE_MODE: "hang" },
policy: probePolicy({
initializeBudgetMs: initializeMs,
firstCatalogBudgetMs: firstCatalogMs,
}),
policy,
});
} catch (cause) {
caught = cause;
}
expect(caught).toBeInstanceOf(Error);
const message = errorMessage(caught);
expect(message).toContain(
`The effective REA_MCP_FIRST_CATALOG_BUDGET_MS value for the MCP first-catalog phase budget is ${String(firstCatalogMs)} ms (source setting; default 20000 ms, floor 1 ms, maximum 2147483647 ms).`,
);
expectPhaseText(message, policy.firstCatalog);
expect(message).not.toContain("REA_MCP_INITIALIZE_BUDGET_MS");
const elapsedMs = performance.now() - started;
expect(elapsedMs).toBeLessThan(initializeMs);
@@ -291,6 +312,10 @@ describe("packaged MCP first-catalog budget while a phase is pending", () => {
const toolsListMs = 8_000;
const root = await createTestTempDirectory("rea-mcp-startup-");
const initializedPath = join(root, "initialized");
const policy = probePolicy({
toolsListBudgetMs: toolsListMs,
firstCatalogBudgetMs: firstCatalogMs,
});
const started = performance.now();
let caught: unknown;
try {
@@ -302,19 +327,14 @@ describe("packaged MCP first-catalog budget while a phase is pending", () => {
REA_MCP_PROBE_MODE: "initialize-then-hang",
REA_MCP_PROBE_INITIALIZED_PATH: initializedPath,
},
policy: probePolicy({
toolsListBudgetMs: toolsListMs,
firstCatalogBudgetMs: firstCatalogMs,
}),
policy,
});
} catch (cause) {
caught = cause;
}
expect(caught).toBeInstanceOf(Error);
const message = errorMessage(caught);
expect(message).toContain(
`The effective REA_MCP_FIRST_CATALOG_BUDGET_MS value for the MCP first-catalog phase budget is ${String(firstCatalogMs)} ms (source setting; default 20000 ms, floor 1 ms, maximum 2147483647 ms).`,
);
expectPhaseText(message, policy.firstCatalog);
expect(message).not.toContain("REA_MCP_TOOLS_LIST_BUDGET_MS");
const elapsedMs = performance.now() - started;
expect(elapsedMs).toBeLessThan(toolsListMs);
@@ -229,49 +229,148 @@ it.each(providerNames)(
},
);
const DIGEST_MISMATCH_REASON =
"Evidence validation failed at evidence_id: Evidence semantic identifier does not match its record";
const identityMismatchReason = (tool: string): string => {
switch (tool) {
case "find_changed_behavior":
return "Comparison Evidence operation, predicate, and provider disagree";
case "correlate_static_and_runtime":
return "Comparison Evidence operation, predicate, provider, or citations disagree";
case "verify_reconstruction":
return "Comparison Evidence identity or authority disagrees";
default:
throw new Error(`Unexpected comparison consumer ${tool}`);
}
};
const unsupportedOperationIssue = (
tool: string,
): {
readonly path: readonly (string | number)[];
readonly message: string;
} => {
switch (tool) {
case "find_changed_behavior":
return {
path: ["comparisons", 0],
message:
"Changed-behavior analysis requires process, artifact, or function comparison Evidence",
};
case "correlate_static_and_runtime":
return {
path: ["runtime_comparisons", 0],
message:
"Static/runtime correlation requires process comparison Evidence",
};
case "verify_reconstruction":
return {
path: ["claims", 0, "comparison_evidence_id"],
message:
"Claim terminal-output requires compare_process_captures Evidence; selected fixture_comparison",
};
default:
throw new Error(`Unexpected comparison consumer ${tool}`);
}
};
const expectComparisonRejection = (
response: Parameters<typeof parseMcpToolError>[0],
tool: string,
kind: "identity" | "operation" | "digest",
) => {
if (kind === "operation") {
const issue = unsupportedOperationIssue(tool);
expect(parseMcpToolError(response), tool).toMatchObject({
error: {
code: "invalid_request",
category: "invalid_input",
message: issue.message,
recovery: { kind: "inspect_input_contract", operation: tool },
details: {
operation: tool,
issues: [
{
path: [...issue.path],
reason: "invalid_value",
message: issue.message,
},
],
},
},
});
return;
}
const reason =
kind === "digest" ? DIGEST_MISMATCH_REASON : identityMismatchReason(tool);
expect(parseMcpToolError(response), tool).toMatchObject({
error: {
code: "evidence_integrity_mismatch",
category: "integrity_mismatch",
message: reason,
details: { reason },
},
});
};
const invalidIdentities: ReadonlyArray<{
label: string;
kind: "identity" | "operation";
change: (base: Evidence) => IdentityChange;
}> = [
{
label: "wrong name",
kind: "identity",
change: ({ provider }) => ({
provider: { ...provider, name: `${provider.name} extra` },
}),
},
{
label: "wrong id",
kind: "identity",
change: ({ provider }) => ({
provider: { ...provider, id: "fixture-process" },
}),
},
{
label: "old version",
kind: "identity",
change: ({ provider }) => ({ provider: { ...provider, version: "2" } }),
},
{
label: "future version",
kind: "identity",
change: ({ provider }) => ({ provider: { ...provider, version: "4" } }),
},
{
label: "missing version",
kind: "identity",
change: ({ provider }) => ({ provider: { ...provider, version: null } }),
},
{
label: "wrong predicate",
kind: "identity",
change: () => ({ predicate_type: "fixture.comparison" }),
},
{
label: "wrong operation",
kind: "operation",
change: () => ({ operation: "fixture_comparison" }),
},
{
label: "non-null subject",
kind: "identity",
change: () => ({ subject: FUNCTION_COMPARISON_EXAMPLE.left.subject }),
},
{ label: "wrong confidence", change: () => ({ confidence: "observed" }) },
{
label: "wrong confidence",
kind: "identity",
change: () => ({ confidence: "observed" }),
},
{
label: "wrong authority",
kind: "identity",
change: () => ({ authority: "controlled-replay" }),
},
];
@@ -279,7 +378,7 @@ const invalidIdentities: ReadonlyArray<{
for (const name of providerNames)
it.each(invalidIdentities)(
`rejects independently rehashed $label for ${name} in every MCP consumer`,
async ({ change }) => {
async ({ change, kind }) => {
const { session, client, comparison } = await connectedComparison();
const named = rehashedComparison(comparison, {
provider: { ...comparison.provider, name },
@@ -291,9 +390,7 @@ for (const name of providerNames)
for (const { expected: _expected, ...call } of consumerCalls(invalid)) {
const response = await client.callTool(call);
expect(response.isError, call.name).toBe(true);
expect(parseMcpToolError(response), call.name).toMatchObject({
error: { code: "invalid_request" },
});
expectComparisonRejection(response, call.name, kind);
}
},
);
@@ -332,13 +429,11 @@ it("separately rejects a tampered digest before admission to any consumer or ses
)) {
const response = await client.callTool(call);
expect(response.isError, call.name).toBe(true);
expect(parseMcpToolError(response)).toMatchObject({
error: { code: "invalid_request" },
});
expectComparisonRejection(response, call.name, "digest");
}
});
it("maps malformed changed-behavior Evidence to an invalid request before closure lookup", async () => {
it("maps malformed changed-behavior Evidence to an integrity mismatch before closure lookup", async () => {
const { client, comparison } = await connectedComparison();
const malformed = { ...comparison, evidence_links: undefined } as unknown;
const response = await client.callTool({
@@ -346,8 +441,15 @@ it("maps malformed changed-behavior Evidence to an invalid request before closur
arguments: { comparisons: [malformed] },
});
expect(response.isError).toBe(true);
const reason =
"Evidence validation failed at evidence_links: Invalid input: expected array, received undefined";
expect(parseMcpToolError(response)).toMatchObject({
error: { code: "invalid_request" },
error: {
code: "evidence_integrity_mismatch",
category: "integrity_mismatch",
message: reason,
details: { reason },
},
});
});
+20 -8
View File
@@ -5,6 +5,10 @@ import { dirname, join } from "node:path";
import { expect, onTestFinished, test, vi } from "vitest";
import { PcapProvider } from "../../../src/pcap/PcapProvider.js";
import {
CAPINFOS_COMMAND_ENV,
TSHARK_COMMAND_ENV,
} from "../../../src/pcap/PcapConfiguration.js";
import { projectAnalysisError } from "../../../src/domain/analysisErrorProjection.js";
import {
pcapResultSchemas,
@@ -419,10 +423,16 @@ it("reports relative configuration and missing capinfos as capability gaps", asy
});
if (result.ok) throw new Error("expected unavailable tools");
const projected = projectAnalysisError(result.error);
expect(projected.code).toBe("capability_unavailable");
expect(projected.message).toContain(
"Supply Wireshark 4.x tshark and capinfos",
);
expect(projected).toMatchObject({
code: "capability_unavailable",
details: { provider_id: "pcap", operation: "inspect_pcap_client" },
});
expect(projected.message).toContain(TSHARK_COMMAND_ENV);
expect(projected.message).toContain(CAPINFOS_COMMAND_ENV);
expect(projected.message).toContain("Wireshark 4.x");
expect(projected.message).toContain("PATH");
expect(projected.message).toContain("absolute");
expect(projected.message).toContain("does not install");
}
});
@@ -443,10 +453,12 @@ it("names the prlimit selection when the Linux limiter is unusable", async () =>
});
if (result.ok) throw new Error("expected an unusable limiter");
const projected = projectAnalysisError(result.error);
expect(projected.code).toBe("capability_unavailable");
expect(projected.message).toContain(
"(REA_TSHARK_PRLIMIT_COMMAND must be absolute.) Check that REA_TSHARK_PRLIMIT_COMMAND",
);
expect(projected).toMatchObject({
code: "capability_unavailable",
details: { provider_id: "pcap", operation: "inspect_pcap_capture" },
});
expect(projected.message).toContain("must be absolute");
expect(projected.message).toContain("Check that REA_TSHARK_PRLIMIT_COMMAND");
});
const nodeTool = async (
@@ -13,6 +13,10 @@ import { expect, it, onTestFinished, vi } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { PcapProvider } from "../../../src/pcap/PcapProvider.js";
import {
CAPINFOS_COMMAND_ENV,
TSHARK_COMMAND_ENV,
} from "../../../src/pcap/PcapConfiguration.js";
import { PcapSnapshot } from "../../../src/pcap/PcapSnapshot.js";
import * as regularFile from "../../../src/filesystem/RegularFile.js";
import { OwnedFileHandle } from "../../../src/filesystem/OwnedFileHandle.js";
@@ -124,12 +128,17 @@ it("projects a missing configured tool as a capability gap", async () => {
});
expect(result.ok).toBe(false);
if (result.ok) return;
expect(projectAnalysisError(result.error)).toMatchObject({
const projected = projectAnalysisError(result.error);
expect(projected).toMatchObject({
code: "capability_unavailable",
message: expect.stringContaining(
"Supply Wireshark 4.x tshark and capinfos",
),
details: { provider_id: "pcap", operation: "inspect_pcap_client" },
});
expect(projected.message).toContain(TSHARK_COMMAND_ENV);
expect(projected.message).toContain(CAPINFOS_COMMAND_ENV);
expect(projected.message).toContain("Wireshark 4.x");
expect(projected.message).toContain("PATH");
expect(projected.message).toContain("absolute");
expect(projected.message).toContain("does not install");
});
it.skipIf(isWindows)(
@@ -302,11 +311,23 @@ it.runIf(!hasNativeBound)(
expect(result.ok).toBe(false);
if (result.ok) return;
const projected = projectAnalysisError(result.error);
expect(projected.code).toBe("capability_unavailable");
expect(projected).toMatchObject({
code: "capability_unavailable",
details: { provider_id: "pcap", operation: "inspect_pcap_capture" },
});
// The refusal concerns this host, so it must not suggest another target.
expect(projected.message).toContain(
"Linux with util-linux prlimit, or Windows with owned Job Objects",
);
const reason =
projected.details &&
"reason" in projected.details &&
typeof projected.details.reason === "string"
? projected.details.reason
: "";
expect(reason).toContain("Linux");
expect(reason).toContain("util-linux");
expect(reason).toContain("prlimit");
expect(reason).toContain("Windows");
expect(reason).toContain("owned");
expect(reason).toContain("Job Objects");
expect(projected.remediation.action).toBe(projected.message);
},
);
+28 -23
View File
@@ -76,21 +76,14 @@ it("names the configurable regex deadline and the effective worker heap", () =>
readLaunchNodeOptions(),
);
expect(REGEX_MATCHING_LIMITATION).toBe(regexMatchingLimitation(heap));
expect(REGEX_MATCHING_LIMITATION).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(REGEX_MATCHING_LIMITATION).toContain(
"5000 ms matching deadline (source default)",
`${String(DEFAULT_REGEX_MATCHING_DEADLINE_MS)} ms (source default`,
);
expect(REGEX_MATCHING_LIMITATION).toContain("REA_REGEX_MATCHING_DEADLINE_MS");
expect(REGEX_MATCHING_LIMITATION).toContain(
"REA_REGEX_WORKER_HEAP_REQUEST_MIB",
);
expect(REGEX_MATCHING_LIMITATION).toContain(
"source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB",
);
expect(REGEX_MATCHING_LIMITATION).toContain("floor 1 ms");
expect(REGEX_MATCHING_LIMITATION).toContain("maximum 2147483647 ms");
expect(REGEX_MATCHING_LIMITATION).not.toContain("no runtime override");
expect(REGEX_MATCHING_LIMITATION).not.toContain("either cap");
expect(REGEX_MATCHING_LIMITATION).not.toContain("or heap");
if (heap.kind === "unknown") {
expect(REGEX_MATCHING_LIMITATION).toContain("was not established");
} else {
@@ -101,14 +94,20 @@ it("names the configurable regex deadline and the effective worker heap", () =>
const deadline = projectAnalysisError(
regexMatchingDeadlineError("search_procedures"),
);
expect(deadline.message).toContain("5000 ms");
expect(deadline.message).toContain("source default");
expect(deadline.message).toContain("REA_REGEX_MATCHING_DEADLINE_MS");
expect(deadline.message).toContain("maximum 2147483647 ms");
expect(deadline).toMatchObject({
code: "resource_constraint",
details: {
resource: "cpu",
reported_limits: {
matching_deadline_ms: DEFAULT_REGEX_MATCHING_DEADLINE_MS,
matching_deadline_source: "default",
},
},
});
expect(deadline.message).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(deadline.remediation.action).toContain(
"REA_REGEX_MATCHING_DEADLINE_MS",
REGEX_MATCHING_DEADLINE_SETTING,
);
expect(deadline.remediation.action).toContain("source default");
expect(deadline.remediation.action).not.toContain("no runtime override");
const failure = projectAnalysisError(
regexWorkerHeapError("search_procedures", new Error("heap")),
@@ -118,9 +117,6 @@ it("names the configurable regex deadline and the effective worker heap", () =>
expect(failure.remediation.action).toContain(
"REA_REGEX_WORKER_HEAP_REQUEST_MIB",
);
expect(failure.remediation.action).toContain(
"source default; default 128 MiB, floor 7 MiB, maximum 16384 MiB",
);
if (heap.kind === "effective") {
expect(failure.message).toContain(`${String(heap.mib)} MiB`);
expect(failure.details).toMatchObject({
@@ -489,15 +485,24 @@ it("stops a slow match at the configured deadline and names the setting", async
expect(matched.ok).toBe(false);
if (matched.ok) return;
const projected = projectAnalysisError(matched.error);
expect(projected.message).toContain("100 ms (source setting)");
expect(projected.message).toContain("REA_REGEX_MATCHING_DEADLINE_MS");
expect(projected.message).toContain("default 5000 ms");
expect(projected.message).toContain("maximum 2147483647 ms");
expect(projected).toMatchObject({
code: "resource_constraint",
details: {
resource: "cpu",
reported_limits: {
matching_deadline_ms: 100,
matching_deadline_source: "setting",
},
},
});
expect(projected.message).toContain(REGEX_MATCHING_DEADLINE_SETTING);
expect(projected.remediation.action).toContain(
"Set REA_REGEX_MATCHING_DEADLINE_MS in the environment that starts REA",
REGEX_MATCHING_DEADLINE_SETTING,
);
expect(projected.remediation.action).not.toContain("no runtime override");
expect(matched.error).toMatchObject({
_tag: "AnalysisResourceConstraintError",
resource: "cpu",
reportedLimits: {
matching_deadline_ms: 100,
matching_deadline_source: "setting",
@@ -46,17 +46,16 @@ try {
assert.equal(matched.ok, false);
assert.equal(matched.error._tag, ${JSON.stringify(mode !== "deadline" ? "AnalysisCancelledError" : "AnalysisResourceConstraintError")});
if (${JSON.stringify(mode)} === "deadline") {
assert.equal(matched.error.resource, "cpu");
assert.equal(matched.error.reportedLimits.matching_deadline_ms, 5000);
assert.equal(matched.error.reportedLimits.matching_deadline_source, "default");
assert.match(matched.error.reason, /5000 ms \\(source default\\)/);
assert.match(matched.error.reason, /REA_REGEX_MATCHING_DEADLINE_MS/);
assert.match(matched.error.reason, /maximum 2147483647 ms/);
assert.match(matched.error.remediationAction, /Set REA_REGEX_MATCHING_DEADLINE_MS in the environment that starts REA/);
assert.equal(matched.error.reason.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(matched.error.remediationAction.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(matched.error.reason.includes("no runtime override"), false);
assert.equal(matched.error.remediationAction.includes("no runtime override"), false);
const limitation = provider.capabilities().find((capability) => capability.operation === ${JSON.stringify(operation)}).limitations.join(" ");
assert.match(limitation, /5000 ms matching deadline \\(source default\\)/);
assert.match(limitation, /REA_REGEX_MATCHING_DEADLINE_MS/);
assert.equal(limitation.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(limitation.includes("source default"), true);
assert.equal(limitation.includes("no runtime override"), false);
}
if (${JSON.stringify(mode)} === "close") {
@@ -115,10 +114,9 @@ const provider = new KunaProvider(${JSON.stringify({
REA_REGEX_MATCHING_DEADLINE_MS: "200",
})});
const limitation = provider.capabilities().find((capability) => capability.operation === "search_procedures").limitations.join(" ");
assert.match(limitation, /200 ms matching deadline \\(source setting\\)/);
assert.match(limitation, /REA_REGEX_MATCHING_DEADLINE_MS/);
assert.match(limitation, /default 5000 ms/);
assert.match(limitation, /maximum 2147483647 ms/);
assert.equal(limitation.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(limitation.includes("200"), true);
assert.equal(limitation.includes("source setting"), true);
assert.equal(limitation.includes("no runtime override"), false);
const client = provider.createClient(${JSON.stringify(target)}, ${JSON.stringify(profile)});
try {
@@ -128,10 +126,12 @@ try {
assert.ok(performance.now() - started < 4000, "deadline stayed at the 5000 ms default");
assert.equal(matched.ok, false);
assert.equal(matched.error._tag, "AnalysisResourceConstraintError");
assert.equal(matched.error.resource, "cpu");
assert.equal(matched.error.reportedLimits.matching_deadline_ms, 200);
assert.equal(matched.error.reportedLimits.matching_deadline_source, "setting");
assert.match(matched.error.reason, /200 ms \\(source setting\\)/);
assert.match(matched.error.remediationAction, /REA_REGEX_MATCHING_DEADLINE_MS/);
assert.equal(matched.error.reason.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(matched.error.remediationAction.includes("REA_REGEX_MATCHING_DEADLINE_MS"), true);
assert.equal(matched.error.reason.includes("source setting"), true);
assert.equal(matched.error.reason.includes("no runtime override"), false);
} finally {
assert.equal((await client.close()).ok, true);