fix(parsers): preserve format evidence and caller input identity
Retain bounded FAT and offset-tagged PE observations through archive hashing, classification, and content nodes. Preserve PyInstaller member paths according to the observed carrier dialect and share APK v1 signing-name semantics. Read SQLite's implicit sqlite_schema table without inventing a stored schema declaration. Distinguish unsupported ELF numbering and decoded-name admission from malformed data and fixed scan allowances. Preserve Sigrok basenames and classify matching explicit input-format refusals at their owning boundary. Validation: 554 distinct tests passed, 3 skipped across serial focused runs; build, typecheck/lint, docs, formatting, and discovery (2141 files) passed. Sigrok command-boundary simulators do not establish real-tool decoding. Small repeated public PyInstaller CLI runs had equivalent inventory facts; warm median elapsed 0.5431s before / 0.5426s after, with no speed claim. Fixes #6733 Fixes #6741 Fixes #6722 Fixes #6732 Fixes #6736 Fixes #6734 Fixes #6764 Fixes #6766 Refs #6739
This commit is contained in:
1 parent
a718037033
commit
bd4f5e9c19
47 files changed
+1523
-209
No files matched your search
@@ -256,6 +256,10 @@ such as
|
||||
`libreactnativejni.so`, `libreactnative.so`, `libhermes.so`,
|
||||
`libhermes_executor.so`, `libflutter.so`, and `libunity.so`; unrelated names
|
||||
such as `libcommunity.so` and `libreactivity.so` do not identify those families.
|
||||
The signing list retains the `META-INF/MANIFEST.MF`, `.SF`, `.RSA`, `.DSA`, and
|
||||
`.EC` paths selected by Android's v1 verifier, including nested paths. These are
|
||||
inventory metadata candidates, not proof that a v1 signature verifies or that
|
||||
the APK is signed; use `inspect-apk-signature` for signature analysis.
|
||||
The bridge basis is inferred from the native path and repeated for each managed
|
||||
component, so it does not identify a specific JNI binding. Bridge hypotheses form a DEX/JVM-class by native-library
|
||||
Cartesian product. The projection computes the product and exact serialized
|
||||
|
||||
+6
-2
@@ -46,13 +46,17 @@ The selected file is limited to 64 MiB by default (512 MiB maximum). At most
|
||||
names, section names, and GNU descriptor text share a scan budget based on
|
||||
the source bytes plus any decoded section-name table, capped at
|
||||
`max_file_bytes`. The raw source and decoded table must also fit together
|
||||
within that ceiling; repeated table ranges count again. Each descriptor range is hashed once,
|
||||
within that ceiling. If admission fails, the error reports the required
|
||||
`max_file_bytes`; raising it can admit the decoded table up to the operation's
|
||||
512 MiB maximum. Once admitted, raising that setting does not increase the
|
||||
scan budget for the same source and decoded names. Repeated table ranges count again. Each descriptor range is hashed once,
|
||||
and each GNU version's trailing NUL run is scanned once; neither walk may
|
||||
exceed that byte ceiling. Malformed ELF headers, truncated header
|
||||
tables, declared note tables that extend past the file, and truncated
|
||||
note entries fail without partial success. ELF extended
|
||||
numbering (`PN_XNUM`, including a section count stored when `e_shnum` is 0) is
|
||||
unsupported.
|
||||
unsupported and returns an unsupported-target outcome rather than invalid-input
|
||||
recovery.
|
||||
|
||||
This workflow does not execute the image or verify GNU build-id uniqueness
|
||||
against package catalogs.
|
||||
@@ -12,8 +12,11 @@ The root occurrence's `format_metadata` records the CArchive layout (2.0 or
|
||||
2.1), archive and cookie offsets, table extent, entry count, raw Python version,
|
||||
normalized major/minor when known, library name and encryption marker. Member
|
||||
metadata retains the original name, table index, type, compression flag,
|
||||
relative offset and known directory record extent. Logical paths use archive
|
||||
path rules; the source name retains its spelling, including backslashes.
|
||||
relative offset and known directory record extent. Logical paths follow the
|
||||
carrier's path dialect: backslashes separate components for PE carriers, while
|
||||
they remain literal filename characters for POSIX carriers. The source name
|
||||
always retains its stored spelling. If the carrier dialect is unconfirmed, REA
|
||||
preserves backslashes verbatim instead of inferring POSIX semantics.
|
||||
`relative_offset` is relative to the CArchive start for CArchive rows, or to
|
||||
the decoded PYZ image for PYZ rows. A CArchive directory record starts at
|
||||
`archive_start + toc_offset + directory_record_offset`; its size is
|
||||
|
||||
@@ -38,12 +38,16 @@ missing or unknown selected channel target, REA returns `invalid_request` for
|
||||
successfully. The diagnostic and any partial stdout remain in `captured_output`;
|
||||
unrelated producer warnings do not change request status.
|
||||
Omitting `input_format` requests sigrok's autodetection. REA uses an immutable
|
||||
private copy whose filename retains the selected recording's final suffix, so
|
||||
filename-based format detection receives that hint without exposing the
|
||||
private copy whose filename retains the selected recording's final suffix,
|
||||
including a dotfile basename such as `.csv`. Filename-based format detection
|
||||
receives that hint without exposing the
|
||||
caller's directory to sigrok. Omitting `annotations`
|
||||
retains every annotation class. Both omitted selections are recorded as null.
|
||||
See the [upstream manual](https://sigrok.org/wiki/Sigrok-cli) for formats,
|
||||
decoder stacking and annotation selectors. No arbitrary command flags are accepted.
|
||||
If sigrok reports an explicitly selected input format as unknown, REA returns
|
||||
`invalid_request` for `input_format`, retains the producer diagnostic, and
|
||||
advises correcting the format or omitting it for autodetection. See the
|
||||
[upstream manual](https://sigrok.org/wiki/Sigrok-cli) for formats, decoder
|
||||
stacking and annotation selectors. No arbitrary command flags are accepted.
|
||||
|
||||
Output defaults to 50 literal LF-separated lines. Use `line_offset` and
|
||||
`line_limit` for later pages, or MCP `line_limit: null` / CLI `--all_lines`
|
||||
|
||||
@@ -54,6 +54,13 @@ reported. If a virtual table cannot be resolved, possible backing tables have
|
||||
ordinary tables remain selectable. Schema inventory does not execute views,
|
||||
triggers or arbitrary SQL.
|
||||
|
||||
The exact table name `sqlite_schema` is also a supported read-only row source.
|
||||
SQLite's [`table_list`](https://www.sqlite.org/pragma.html#pragma_table_list)
|
||||
and `table_xinfo` pragmas establish its row-source type and columns. The
|
||||
[schema catalog](https://www.sqlite.org/schematab.html) has no self-entry for
|
||||
`sqlite_schema`, so REA does not invent a declaration, SQL statement or root
|
||||
page for it in `schema.tables`.
|
||||
|
||||
Rows are optional. `table` selects one exact ordinary-table name. `columns`
|
||||
(CLI repeated `--column`) selects exact visible column names in the requested
|
||||
order; omission selects all visible columns, including generated columns.
|
||||
|
||||
@@ -574,7 +574,9 @@ const readExecutableMetadata = async (
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (bytes.length >= 2 && bytes[0] === 0x4d && bytes[1] === 0x5a) {
|
||||
let evidence = parseMzHeaderEvidence(bytes, fileSize);
|
||||
let signatureBytes: Buffer | undefined;
|
||||
let signatureEvidence:
|
||||
| { readonly offset: number; readonly bytes: Buffer }
|
||||
| undefined;
|
||||
if (evidence.windowsHeader.state === "signature-needed") {
|
||||
const { offset } = evidence.windowsHeader;
|
||||
const signature = Buffer.alloc(Math.min(4, fileSize - offset));
|
||||
@@ -585,8 +587,11 @@ const readExecutableMetadata = async (
|
||||
offset,
|
||||
);
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
signatureBytes = signature.subarray(0, observedSignature.bytesRead);
|
||||
evidence = parseMzHeaderEvidence(bytes, fileSize, signatureBytes);
|
||||
signatureEvidence = {
|
||||
offset,
|
||||
bytes: signature.subarray(0, observedSignature.bytesRead),
|
||||
};
|
||||
evidence = parseMzHeaderEvidence(bytes, fileSize, signatureEvidence);
|
||||
}
|
||||
const { windowsHeader } = evidence;
|
||||
if (windowsHeader.state === "declared") {
|
||||
@@ -618,7 +623,7 @@ const readExecutableMetadata = async (
|
||||
evidence = parseMzHeaderEvidence(
|
||||
table.subarray(0, observed.bytesRead),
|
||||
fileSize,
|
||||
signatureBytes,
|
||||
signatureEvidence,
|
||||
);
|
||||
if (evidence.windowsHeader.state === "candidate")
|
||||
return err(
|
||||
@@ -628,6 +633,7 @@ const readExecutableMetadata = async (
|
||||
table.subarray(0, observed.bytesRead),
|
||||
hostArchitecture,
|
||||
fileSize,
|
||||
signatureEvidence,
|
||||
);
|
||||
}
|
||||
if (windowsHeader.state === "candidate")
|
||||
@@ -636,7 +642,7 @@ const readExecutableMetadata = async (
|
||||
bytes,
|
||||
hostArchitecture,
|
||||
fileSize,
|
||||
signatureBytes,
|
||||
signatureEvidence,
|
||||
);
|
||||
}
|
||||
const fatEvidence = await readFatSliceHeaderEvidence(
|
||||
@@ -797,11 +803,10 @@ const readPeMetadata = async (
|
||||
throwIfTargetResolutionCancelled(signal);
|
||||
if (fileHeaderRead.bytesRead !== fileHeader.length)
|
||||
return err("invalid or truncated PE header");
|
||||
const windowsHeader = observeMzWindowsHeader(
|
||||
mzPrefix,
|
||||
fileSize,
|
||||
fileHeader.subarray(0, 4),
|
||||
);
|
||||
const windowsHeader = observeMzWindowsHeader(mzPrefix, fileSize, {
|
||||
offset,
|
||||
bytes: fileHeader.subarray(0, 4),
|
||||
});
|
||||
if (windowsHeader.state !== "declared")
|
||||
return err("invalid Windows new-header signature in MZ image");
|
||||
if (["NE", "LE", "LX"].includes(windowsHeader.signature))
|
||||
|
||||
@@ -122,6 +122,90 @@ it("retains the actual default without inventing an explicit caller selection",
|
||||
expect(result.value.parameters).not.toHaveProperty("worker_timeout_ms");
|
||||
});
|
||||
|
||||
it("accepts sqlite_schema rows without requiring a fabricated declaration", async () => {
|
||||
const report = {
|
||||
...fixture(),
|
||||
schema: {
|
||||
...fixture().schema,
|
||||
scope: "selected_table",
|
||||
tables: [],
|
||||
},
|
||||
rows: {
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
values: [
|
||||
[
|
||||
{
|
||||
type: "text",
|
||||
value: "records",
|
||||
bytes_hex: "7265636f726473",
|
||||
encoding: "UTF-8",
|
||||
},
|
||||
{
|
||||
type: "text",
|
||||
value: "table",
|
||||
bytes_hex: "7461626c65",
|
||||
encoding: "UTF-8",
|
||||
},
|
||||
],
|
||||
],
|
||||
row_limit: 20,
|
||||
returned_rows: 1,
|
||||
truncated: false,
|
||||
order: "unspecified",
|
||||
},
|
||||
};
|
||||
const service = new SqliteDatabaseService({
|
||||
inspect: async () => ok(execution(report)),
|
||||
});
|
||||
const result = await service.inspect({
|
||||
path: "/selected.db",
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
schema_scope: "selected_table",
|
||||
});
|
||||
expect(result.ok).toBe(true);
|
||||
|
||||
const foreign = {
|
||||
...report,
|
||||
schema: {
|
||||
...report.schema,
|
||||
tables: [
|
||||
{
|
||||
name: "records",
|
||||
root_page: 2,
|
||||
kind: "table",
|
||||
without_rowid: false,
|
||||
strict: false,
|
||||
column_count: 0,
|
||||
columns_completeness: "complete",
|
||||
columns: [],
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
const foreignService = new SqliteDatabaseService({
|
||||
inspect: async () => ok(execution(foreign)),
|
||||
});
|
||||
expect(
|
||||
await foreignService.inspect({
|
||||
path: "/selected.db",
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
schema_scope: "selected_table",
|
||||
}),
|
||||
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
|
||||
|
||||
expect(
|
||||
await service.inspect({
|
||||
path: "/selected.db",
|
||||
table: "missing",
|
||||
columns: ["name", "type"],
|
||||
schema_scope: "selected_table",
|
||||
}),
|
||||
).toMatchObject({ ok: false, error: { _tag: "AnalysisOutputError" } });
|
||||
});
|
||||
|
||||
it("parses historical report omissions as unknown without supplying a default", () => {
|
||||
const { worker_timeout_ms: _default, ...historical } = fixture();
|
||||
const parsed = sqliteDatabaseSchema.parse(historical);
|
||||
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
type SqliteDatabase,
|
||||
SQLITE_ROW_LIMIT_DEFAULT,
|
||||
DEFAULT_SQLITE_WORKER_TIMEOUT_MS,
|
||||
sqliteSchemaHasObjectDeclaration,
|
||||
sqliteSchemaScope,
|
||||
sqliteInspectionParameters,
|
||||
} from "../../domain/sqlite/sqliteDatabase.js";
|
||||
@@ -26,6 +27,21 @@ import { jsonValueSchema } from "../../domain/jsonValue.js";
|
||||
|
||||
const OPERATION = "inspect_sqlite_database";
|
||||
|
||||
const matchesSelectedSchema = (
|
||||
schema: SqliteDatabase["schema"],
|
||||
table: string,
|
||||
): boolean =>
|
||||
sqliteSchemaHasObjectDeclaration(table)
|
||||
? schema.tables.length === 1 &&
|
||||
schema.tables[0]?.name === table &&
|
||||
schema.indexes.every((index) => index.table === table) &&
|
||||
schema.triggers.every((trigger) => trigger.table === table) &&
|
||||
schema.views.length === 0
|
||||
: schema.tables.length === 0 &&
|
||||
schema.indexes.length === 0 &&
|
||||
schema.views.length === 0 &&
|
||||
schema.triggers.length === 0;
|
||||
|
||||
const matchesSelectedDatabase = (
|
||||
report: SqliteDatabase,
|
||||
input: InspectSqliteDatabaseInput,
|
||||
@@ -41,13 +57,8 @@ const matchesSelectedDatabase = (
|
||||
(report.wal === null || report.wal.path === `${input.path}-wal`) &&
|
||||
report.schema.scope === sqliteSchemaScope(input) &&
|
||||
(report.schema.scope !== "selected_table" ||
|
||||
(report.schema.tables.length === 1 &&
|
||||
report.schema.tables[0]?.name === input.table &&
|
||||
report.schema.indexes.every((index) => index.table === input.table) &&
|
||||
report.schema.triggers.every(
|
||||
(trigger) => trigger.table === input.table,
|
||||
) &&
|
||||
report.schema.views.length === 0)) &&
|
||||
(input.table !== undefined &&
|
||||
matchesSelectedSchema(report.schema, input.table))) &&
|
||||
(input.table === undefined
|
||||
? report.rows === null
|
||||
: report.rows?.table === input.table &&
|
||||
|
||||
@@ -15,6 +15,10 @@ import {
|
||||
type MachoHeaderPrefixFacts,
|
||||
type FatSliceHeaderEvidence,
|
||||
} from "../domain/apple/machoContainer.js";
|
||||
import {
|
||||
observeMzWindowsHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "../domain/dosMz.js";
|
||||
|
||||
/** Initial bytes required by the other bounded format classifiers. */
|
||||
export const ARTIFACT_CLASSIFICATION_PREFIX_BYTES = 8_192;
|
||||
@@ -24,6 +28,7 @@ export interface ArtifactClassificationEvidence {
|
||||
readonly prefix: Buffer;
|
||||
readonly asarHeader: FinalAsarHeaderProbeObservation;
|
||||
readonly fatSliceHeaders?: FatSliceHeaderEvidence;
|
||||
readonly mzWindowsHeaderSignature?: MzWindowsHeaderSignatureEvidence;
|
||||
}
|
||||
|
||||
const observeFatSliceHeaders = (input: {
|
||||
@@ -74,6 +79,7 @@ const finishClassificationEvidence = (
|
||||
archiveBytes: number,
|
||||
finishAsar: (archiveBytes: number) => FinalAsarHeaderProbeObservation,
|
||||
sliceHeadersByOffset: Map<number, MachoHeaderPrefixFacts>,
|
||||
mzWindowsHeaderSignature: MzWindowsHeaderSignatureEvidence | undefined,
|
||||
): ArtifactClassificationEvidence => {
|
||||
const finalHeader = readFatContainerHeader(prefix, archiveBytes);
|
||||
if (finalHeader.state === "parsed") {
|
||||
@@ -102,6 +108,9 @@ const finishClassificationEvidence = (
|
||||
prefix,
|
||||
asarHeader: finishAsar(archiveBytes),
|
||||
...(fatSliceHeaders === undefined ? {} : { fatSliceHeaders }),
|
||||
...(mzWindowsHeaderSignature === undefined
|
||||
? {}
|
||||
: { mzWindowsHeaderSignature }),
|
||||
};
|
||||
};
|
||||
|
||||
@@ -120,6 +129,37 @@ export const createArtifactClassificationProbe = (
|
||||
number,
|
||||
{ readonly bytes: Buffer; count: number }
|
||||
>();
|
||||
let mzWindowsHeaderSignature: MzWindowsHeaderSignatureEvidence | undefined;
|
||||
let mzWindowsHeaderOffset: number | undefined;
|
||||
let mzHeaderChecked = false;
|
||||
const observeSignatureWindow = (
|
||||
offset: number,
|
||||
sourceOffset: number,
|
||||
sourceBytes: Uint8Array,
|
||||
): void => {
|
||||
if (!Number.isSafeInteger(offset) || offset < 0) return;
|
||||
const start = Math.max(offset, sourceOffset);
|
||||
const end = Math.min(offset + 4, sourceOffset + sourceBytes.length);
|
||||
if (start >= end) return;
|
||||
const window = Buffer.alloc(4);
|
||||
const previous =
|
||||
mzWindowsHeaderSignature?.offset === offset
|
||||
? mzWindowsHeaderSignature.bytes
|
||||
: undefined;
|
||||
previous?.copy(window);
|
||||
let observedLength = previous?.length ?? 0;
|
||||
if (start < end && start <= offset + observedLength) {
|
||||
Buffer.from(
|
||||
sourceBytes.subarray(start - sourceOffset, end - sourceOffset),
|
||||
).copy(window, start - offset);
|
||||
observedLength = Math.max(observedLength, end - offset);
|
||||
}
|
||||
if (observedLength === 0) return;
|
||||
mzWindowsHeaderSignature = {
|
||||
offset,
|
||||
bytes: window.subarray(0, observedLength),
|
||||
};
|
||||
};
|
||||
const retain = (bytes: Uint8Array): void => {
|
||||
let offset = 0;
|
||||
while (offset < bytes.length) {
|
||||
@@ -193,6 +233,31 @@ export const createArtifactClassificationProbe = (
|
||||
streamOffset,
|
||||
bytes,
|
||||
});
|
||||
if (!mzHeaderChecked && retainedBytes >= 64) {
|
||||
mzHeaderChecked = true;
|
||||
const prefix = retainedPrefix();
|
||||
if (prefix[0] === 0x4d && prefix[1] === 0x5a) {
|
||||
const windowsHeader = observeMzWindowsHeader(
|
||||
prefix,
|
||||
Number.MAX_SAFE_INTEGER,
|
||||
);
|
||||
if (windowsHeader.state === "signature-needed") {
|
||||
mzWindowsHeaderOffset = windowsHeader.offset;
|
||||
if (windowsHeader.offset < prefix.length) {
|
||||
const inline = prefix.subarray(
|
||||
windowsHeader.offset,
|
||||
Math.min(windowsHeader.offset + 4, prefix.length),
|
||||
);
|
||||
mzWindowsHeaderSignature = {
|
||||
offset: windowsHeader.offset,
|
||||
bytes: inline,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (mzWindowsHeaderOffset !== undefined)
|
||||
observeSignatureWindow(mzWindowsHeaderOffset, streamOffset, bytes);
|
||||
observedBytes += bytes.length;
|
||||
},
|
||||
finish(archiveBytes: number): ArtifactClassificationEvidence {
|
||||
@@ -201,6 +266,7 @@ export const createArtifactClassificationProbe = (
|
||||
archiveBytes,
|
||||
(bytes) => asar.finish(bytes),
|
||||
fatSliceHeadersByOffset,
|
||||
mzWindowsHeaderSignature,
|
||||
);
|
||||
},
|
||||
};
|
||||
|
||||
@@ -31,6 +31,12 @@ import { abortIfNeeded, hashReadable } from "./ArtifactHash.js";
|
||||
import { readArtifactFileChunks } from "./readArtifactFileChunks.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { artifactIntegrityDetails } from "../domain/artifactIntegrity.js";
|
||||
import { parseMzHeaderEvidence } from "../domain/dosMz.js";
|
||||
import {
|
||||
readFatContainerHeader,
|
||||
readMachoHeaderPrefixFacts,
|
||||
} from "../domain/apple/machoContainer.js";
|
||||
import { ARTIFACT_CLASSIFICATION_PREFIX_BYTES } from "./ArtifactClassificationProbe.js";
|
||||
import { PyInstallerPyzStore } from "./pyinstaller/PyInstallerPyzStore.js";
|
||||
import { pyInstallerMemberStream } from "./pyinstaller/PyInstallerMemberStream.js";
|
||||
import {
|
||||
@@ -47,10 +53,12 @@ import {
|
||||
sealMember,
|
||||
type PyInstallerMember,
|
||||
type PyzImage,
|
||||
type CarrierPathDialect,
|
||||
} from "./pyinstaller/CArchiveMembers.js";
|
||||
import { readPyzArchive } from "./pyinstaller/PyzArchive.js";
|
||||
import type { PyInstallerFailure } from "./pyinstaller/PyInstallerFailure.js";
|
||||
import { pyInstallerFileAdmissionFailure } from "./pyinstaller/PyInstallerFileAdmission.js";
|
||||
import { readPyInstallerBytes } from "./pyinstaller/PyInstallerBytes.js";
|
||||
|
||||
const MAX_PARSED_FACT_BYTES = 32 * 1024 * 1024;
|
||||
const MAX_RETAINED_ENTRY_BYTES = 64 * 1024 * 1024;
|
||||
@@ -59,6 +67,45 @@ const RECORD_OWNERSHIP_RESERVATION = 256;
|
||||
// Repeated prefixes are charged again rather than underestimating retained work.
|
||||
const PATH_COMPONENT_RESERVATION = 1024;
|
||||
|
||||
const carrierPathDialect = async (
|
||||
handle: FileHandle,
|
||||
fileSize: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<CarrierPathDialect> => {
|
||||
const prefixRead = await readPyInstallerBytes(handle, {
|
||||
position: 0,
|
||||
length: Math.min(fileSize, ARTIFACT_CLASSIFICATION_PREFIX_BYTES),
|
||||
label: "carrier header",
|
||||
signal,
|
||||
});
|
||||
if (!prefixRead.ok) return "unknown";
|
||||
const prefix = prefixRead.value;
|
||||
let windowsHeader = parseMzHeaderEvidence(prefix, fileSize).windowsHeader;
|
||||
if (windowsHeader.state === "signature-needed") {
|
||||
const signatureRead = await readPyInstallerBytes(handle, {
|
||||
position: windowsHeader.offset,
|
||||
length: 4,
|
||||
label: "PE signature",
|
||||
signal,
|
||||
});
|
||||
if (!signatureRead.ok) return "unknown";
|
||||
windowsHeader = parseMzHeaderEvidence(prefix, fileSize, {
|
||||
offset: windowsHeader.offset,
|
||||
bytes: signatureRead.value,
|
||||
}).windowsHeader;
|
||||
}
|
||||
if (windowsHeader.state === "declared")
|
||||
return windowsHeader.signature === "PE" ? "windows" : "unknown";
|
||||
if (prefix.subarray(0, 4).equals(Buffer.from([0x7f, 0x45, 0x4c, 0x46])))
|
||||
return "posix";
|
||||
const thinMachO = readMachoHeaderPrefixFacts(prefix);
|
||||
if (thinMachO.identity !== null) return "posix";
|
||||
if (readFatContainerHeader(prefix, fileSize).state === "parsed")
|
||||
return "posix";
|
||||
// A missing carrier proof does not establish POSIX separator semantics.
|
||||
return "unknown";
|
||||
};
|
||||
|
||||
interface LoadedArchive {
|
||||
readonly members: readonly PyInstallerMember[];
|
||||
readonly traversal: ArtifactReaderTraversal;
|
||||
@@ -301,6 +348,7 @@ export class PyInstallerArtifactReader implements ArtifactReader {
|
||||
const handle = await this.#openHandle(signal);
|
||||
await this.#verifySource();
|
||||
const size = (await handle.stat()).size;
|
||||
const pathDialect = await carrierPathDialect(handle, size, signal);
|
||||
const cookie = await this.#cookie(handle, size, signal);
|
||||
this.#budget = artifactExpansionBudgetBytes(cookie.packageLength);
|
||||
const toc = await this.#toc(handle, cookie, signal);
|
||||
@@ -320,6 +368,7 @@ export class PyInstallerArtifactReader implements ArtifactReader {
|
||||
images,
|
||||
magic,
|
||||
encrypted,
|
||||
pathDialect,
|
||||
signal,
|
||||
);
|
||||
for (const member of members)
|
||||
@@ -478,6 +527,7 @@ export class PyInstallerArtifactReader implements ArtifactReader {
|
||||
images: ReadonlyMap<CArchiveEntry, PyzImage>,
|
||||
magic: Buffer | undefined,
|
||||
encrypted: boolean,
|
||||
pathDialect: CarrierPathDialect,
|
||||
signal: AbortSignal | undefined,
|
||||
): Promise<PyInstallerMember[]> {
|
||||
const members: PyInstallerMember[] = [];
|
||||
@@ -503,10 +553,20 @@ export class PyInstallerArtifactReader implements ArtifactReader {
|
||||
abortIfNeeded(signal);
|
||||
const image = entry.typeCode === "z" ? images.get(entry) : undefined;
|
||||
if (image === undefined) {
|
||||
append(projectStoredMember(entry, cookie.archiveStart, magic));
|
||||
append(
|
||||
projectStoredMember(entry, cookie.archiveStart, magic, pathDialect),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
append(projectPyzContainer(entry, cookie.archiveStart, image, encrypted));
|
||||
append(
|
||||
projectPyzContainer(
|
||||
entry,
|
||||
cookie.archiveStart,
|
||||
image,
|
||||
encrypted,
|
||||
pathDialect,
|
||||
),
|
||||
);
|
||||
if (image.archive.kind !== "entries") continue;
|
||||
for (const item of image.archive.entries) {
|
||||
if (members.length % 256 === 0) await setImmediate();
|
||||
@@ -517,6 +577,7 @@ export class PyInstallerArtifactReader implements ArtifactReader {
|
||||
item,
|
||||
encrypted,
|
||||
image.archive.pythonMagic,
|
||||
pathDialect,
|
||||
item.namespace || encrypted
|
||||
? undefined
|
||||
: {
|
||||
|
||||
@@ -15,10 +15,35 @@ import {
|
||||
} from "./ArtifactGraphConstruction.js";
|
||||
import { createArtifactClassificationProbe } from "../ArtifactClassificationProbe.js";
|
||||
import { artifactIdForContent } from "../../domain/artifactIdentity.js";
|
||||
import { pe } from "../../domain/binaryTarget.fixture.js";
|
||||
import type { ArtifactEntry } from "../ArtifactReader.js";
|
||||
|
||||
import { classifyArtifactBytesAtPath } from "../../../tests/fixtures/artifactClassification.js";
|
||||
|
||||
it("joins only observed PE signature bytes across stream chunks", () => {
|
||||
const bytes = pe(0x8664, 8190);
|
||||
const probe = createArtifactClassificationProbe();
|
||||
probe.write(bytes.subarray(0, 8191));
|
||||
probe.write(bytes.subarray(8191, 8192));
|
||||
const partial = probe.finish(bytes.length);
|
||||
expect(partial.mzWindowsHeaderSignature).toEqual({
|
||||
offset: 8190,
|
||||
bytes: Buffer.from("PE"),
|
||||
});
|
||||
expect(
|
||||
classifyArtifactContent("native.exe", partial, bytes.length).format,
|
||||
).toBe("unknown");
|
||||
probe.write(bytes.subarray(8192));
|
||||
const complete = probe.finish(bytes.length);
|
||||
expect(complete.mzWindowsHeaderSignature).toEqual({
|
||||
offset: 8190,
|
||||
bytes: Buffer.from([0x50, 0x45, 0, 0]),
|
||||
});
|
||||
expect(
|
||||
classifyArtifactContent("native.exe", complete, bytes.length).format,
|
||||
).toBe("pe");
|
||||
});
|
||||
|
||||
it("keeps an ambiguous DOS MZ inventory candidate unknown", () => {
|
||||
const bytes = Buffer.alloc(128);
|
||||
bytes.write("MZ", 0, "ascii");
|
||||
@@ -358,6 +383,18 @@ describe("bounded FAT inventory classification", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps byte content format separate from path-specific package role", () => {
|
||||
const probe = createArtifactClassificationProbe();
|
||||
probe.write(Buffer.from("PK\x03\x04", "binary"));
|
||||
expect(
|
||||
classifyArtifactContent("application.apk", probe.finish(4), 4),
|
||||
).toEqual({
|
||||
kind: "container",
|
||||
format: "apk",
|
||||
contentFormat: "zip",
|
||||
});
|
||||
});
|
||||
|
||||
describe("FAT prefix observation coverage", () => {
|
||||
it("does not invent an observation for a header beyond the retained prefix", () => {
|
||||
const bytes = Buffer.alloc(8_224);
|
||||
|
||||
@@ -189,12 +189,12 @@ export const materializeDirectoryNodes = (
|
||||
};
|
||||
|
||||
export const createRootNode = (input: {
|
||||
readonly digest: {
|
||||
readonly sha256: string;
|
||||
readonly bytes: number;
|
||||
readonly prefix: Buffer;
|
||||
readonly fatSliceHeaders?: ArtifactClassificationEvidence["fatSliceHeaders"];
|
||||
} | null;
|
||||
readonly digest:
|
||||
| ({
|
||||
readonly sha256: string;
|
||||
readonly bytes: number;
|
||||
} & ArtifactClassificationEvidence)
|
||||
| null;
|
||||
readonly occurrences: readonly MutableOccurrence[];
|
||||
}): ArtifactNode =>
|
||||
createArtifactNode({
|
||||
@@ -218,11 +218,7 @@ export const createRootNode = (input: {
|
||||
format:
|
||||
input.digest === null
|
||||
? "directory"
|
||||
: classifyArtifactBytes(
|
||||
input.digest.prefix,
|
||||
input.digest.bytes,
|
||||
input.digest.fatSliceHeaders,
|
||||
),
|
||||
: classifyArtifactBytes(input.digest, input.digest.bytes),
|
||||
contentState: "materialized",
|
||||
});
|
||||
|
||||
@@ -510,28 +506,31 @@ export const classifyArtifactContent = (
|
||||
): {
|
||||
readonly kind: ArtifactOccurrence["artifact_kind"];
|
||||
readonly format: ArtifactOccurrence["artifact_format"];
|
||||
readonly contentFormat: ArtifactNode["format"];
|
||||
} => {
|
||||
const { prefix, asarHeader, fatSliceHeaders } = evidence;
|
||||
const format = classifyArtifactBytes(prefix, fileSize, fatSliceHeaders);
|
||||
const { prefix, asarHeader } = evidence;
|
||||
const contentFormat = classifyArtifactBytes(evidence, fileSize);
|
||||
const byPath = classifyArtifactPath(path);
|
||||
let role: ReturnType<typeof classifyArtifactPath>;
|
||||
if (
|
||||
format === "plist" &&
|
||||
contentFormat === "plist" &&
|
||||
!hasBinaryPlistSignature(prefix) &&
|
||||
(["javascript", "source-map", "entitlements"].includes(byPath.kind) ||
|
||||
path.toLowerCase().endsWith(".asar"))
|
||||
)
|
||||
return byPath;
|
||||
if (
|
||||
format === "file" &&
|
||||
role = byPath;
|
||||
else if (
|
||||
contentFormat === "file" &&
|
||||
path.toLowerCase().endsWith(".asar") &&
|
||||
asarHeader.state !== "rejected"
|
||||
) {
|
||||
if (asarHeader.state === "limit")
|
||||
throw asarHeaderAdmissionFailure(asarHeader.admission);
|
||||
if (asarHeader.state === "matched")
|
||||
return { kind: "container", format: "asar" };
|
||||
}
|
||||
return artifactRoleForFormat(path, format);
|
||||
role = { kind: "container", format: "asar" };
|
||||
else role = artifactRoleForFormat(path, contentFormat);
|
||||
} else role = artifactRoleForFormat(path, contentFormat);
|
||||
return { ...role, contentFormat };
|
||||
};
|
||||
|
||||
const artifactRoleForFormat = (
|
||||
@@ -558,15 +557,15 @@ const artifactRoleForFormat = (
|
||||
|
||||
/** Classify storage bytes independently of path names, roles, and permissions. */
|
||||
export const classifyArtifactBytes = (
|
||||
prefix: Buffer,
|
||||
fileSize = prefix.length,
|
||||
fatSliceHeaders?: ArtifactClassificationEvidence["fatSliceHeaders"],
|
||||
evidence: ArtifactClassificationEvidence,
|
||||
fileSize = evidence.prefix.length,
|
||||
): ArtifactNode["format"] => {
|
||||
const { prefix, mzWindowsHeaderSignature } = evidence;
|
||||
if (hasZipSignature(prefix)) return "zip";
|
||||
if (hasGodotPckHeader(prefix)) return "godot-pck";
|
||||
if (hasBinaryPlistSignature(prefix)) return "plist";
|
||||
if (hasXmlPlistSignature(prefix)) return "plist";
|
||||
const fatFormat = classifyFatArtifact(prefix, fileSize, fatSliceHeaders);
|
||||
const fatFormat = classifyFatArtifact(evidence, fileSize);
|
||||
if (fatFormat !== null) return fatFormat;
|
||||
if (prefix.length >= 4) {
|
||||
const magic = prefix.readUInt32BE(0);
|
||||
@@ -582,8 +581,12 @@ export const classifyArtifactBytes = (
|
||||
prefix.length < Math.min(fileSize, 64)
|
||||
)
|
||||
return "unknown";
|
||||
const evidence = parseMzHeaderEvidence(prefix, fileSize);
|
||||
const { windowsHeader } = evidence;
|
||||
const mzEvidence = parseMzHeaderEvidence(
|
||||
prefix,
|
||||
fileSize,
|
||||
mzWindowsHeaderSignature,
|
||||
);
|
||||
const { windowsHeader } = mzEvidence;
|
||||
if (windowsHeader.state === "signature-needed") return "unknown";
|
||||
if (windowsHeader.state === "candidate") return "unknown";
|
||||
if (windowsHeader.state === "declared") {
|
||||
@@ -591,22 +594,24 @@ export const classifyArtifactBytes = (
|
||||
if (
|
||||
windowsHeader.signature === "PE" &&
|
||||
windowsOffset >= 64 &&
|
||||
windowsOffset <= prefix.length - 4
|
||||
(windowsOffset <= prefix.length - 4 ||
|
||||
(mzWindowsHeaderSignature?.offset === windowsOffset &&
|
||||
mzWindowsHeaderSignature.bytes.length >= 4))
|
||||
)
|
||||
return "pe";
|
||||
return "unknown";
|
||||
}
|
||||
if (evidence.dos.ok) return "dos-mz";
|
||||
if (mzEvidence.dos.ok) return "dos-mz";
|
||||
return "unknown";
|
||||
}
|
||||
return "file";
|
||||
};
|
||||
|
||||
const classifyFatArtifact = (
|
||||
prefix: Buffer,
|
||||
evidence: ArtifactClassificationEvidence,
|
||||
fileSize: number,
|
||||
fatSliceHeaders?: ArtifactClassificationEvidence["fatSliceHeaders"],
|
||||
): ArtifactNode["format"] | null => {
|
||||
const { prefix, fatSliceHeaders } = evidence;
|
||||
const fatHeader =
|
||||
fatSliceHeaders?.header ?? readFatContainerHeader(prefix, fileSize);
|
||||
if (fatHeader.state === "needs-data" || fatHeader.state === "malformed")
|
||||
|
||||
@@ -26,8 +26,12 @@ import {
|
||||
openRootArtifact,
|
||||
} from "./hashStableRootArtifact.js";
|
||||
import type { ArtifactInventoryOptions } from "./types.js";
|
||||
import type { FatSliceHeaderEvidence } from "../../domain/apple/machoContainer.js";
|
||||
import { readMachoHeaderPrefixFacts } from "../../domain/apple/machoContainer.js";
|
||||
import {
|
||||
observeMzWindowsHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "../../domain/dosMz.js";
|
||||
import type { ArtifactClassificationEvidence } from "../ArtifactClassificationProbe.js";
|
||||
|
||||
interface RootClassification {
|
||||
readonly format: ArtifactOccurrence["artifact_format"];
|
||||
@@ -186,17 +190,20 @@ const classifyRootFormat = async (
|
||||
return "dmg";
|
||||
if (await locateGodotPck(handle, archiveBytes, signal, evidence.prefix))
|
||||
return "godot-pck";
|
||||
const fatSliceHeaders = await readRootFatSliceHeaders(
|
||||
const fatEvidence = await readRootFatSliceHeaders(handle, evidence, signal);
|
||||
const mzWindowsHeaderSignature = await readRootMzWindowsHeaderSignature(
|
||||
handle,
|
||||
evidence.fatSliceHeaders,
|
||||
fatEvidence,
|
||||
archiveBytes,
|
||||
signal,
|
||||
);
|
||||
const classificationEvidence =
|
||||
mzWindowsHeaderSignature === undefined
|
||||
? fatEvidence
|
||||
: { ...fatEvidence, mzWindowsHeaderSignature };
|
||||
const format = classifyArtifactContent(
|
||||
path,
|
||||
{
|
||||
...evidence,
|
||||
...(fatSliceHeaders === undefined ? {} : { fatSliceHeaders }),
|
||||
},
|
||||
classificationEvidence,
|
||||
archiveBytes,
|
||||
).format;
|
||||
// Only bytes no other format claims may lead a located ZIP, so an
|
||||
@@ -228,10 +235,11 @@ const classifyRootFormat = async (
|
||||
/** Read exact FAT header windows missed by the bounded prefix probe. */
|
||||
const readRootFatSliceHeaders = async (
|
||||
handle: FileHandle,
|
||||
existing: FatSliceHeaderEvidence | undefined,
|
||||
evidence: ArtifactClassificationEvidence,
|
||||
signal?: AbortSignal,
|
||||
): Promise<FatSliceHeaderEvidence | undefined> => {
|
||||
if (existing === undefined) return undefined;
|
||||
): Promise<ArtifactClassificationEvidence> => {
|
||||
const existing = evidence.fatSliceHeaders;
|
||||
if (existing === undefined) return evidence;
|
||||
const { header } = existing;
|
||||
const sliceHeadersByOffset = new Map(existing.sliceHeadersByOffset);
|
||||
for (let index = 0; index < header.slices.length; index += 1) {
|
||||
@@ -265,7 +273,59 @@ const readRootFatSliceHeaders = async (
|
||||
readMachoHeaderPrefixFacts(thinHeader.subarray(0, bytesRead)),
|
||||
);
|
||||
}
|
||||
return { header, sliceHeadersByOffset };
|
||||
return { ...evidence, fatSliceHeaders: { header, sliceHeadersByOffset } };
|
||||
};
|
||||
|
||||
/** Read only the missing signature bytes requested by the bounded MZ parser. */
|
||||
const readRootMzWindowsHeaderSignature = async (
|
||||
handle: FileHandle,
|
||||
evidence: ArtifactClassificationEvidence,
|
||||
fileSize: number,
|
||||
signal?: AbortSignal,
|
||||
): Promise<MzWindowsHeaderSignatureEvidence | undefined> => {
|
||||
const observation = observeMzWindowsHeader(evidence.prefix, fileSize);
|
||||
if (observation.state !== "signature-needed")
|
||||
return evidence.mzWindowsHeaderSignature;
|
||||
const { offset } = observation;
|
||||
if (!Number.isSafeInteger(offset) || offset < 0 || offset >= fileSize)
|
||||
return evidence.mzWindowsHeaderSignature;
|
||||
|
||||
const window = Buffer.alloc(4);
|
||||
const existing =
|
||||
evidence.mzWindowsHeaderSignature?.offset === offset
|
||||
? evidence.mzWindowsHeaderSignature.bytes
|
||||
: undefined;
|
||||
const inline =
|
||||
offset < evidence.prefix.length
|
||||
? evidence.prefix.subarray(
|
||||
offset,
|
||||
Math.min(offset + 4, evidence.prefix.length),
|
||||
)
|
||||
: undefined;
|
||||
const known =
|
||||
existing !== undefined &&
|
||||
(inline === undefined || existing.length > inline.length)
|
||||
? existing
|
||||
: inline;
|
||||
known?.copy(window);
|
||||
let available = known?.length ?? 0;
|
||||
const requiredLength = (): number =>
|
||||
available >= 2 && window[0] === 0x50 && window[1] === 0x45 ? 4 : 2;
|
||||
while (available < requiredLength() && offset + available < fileSize) {
|
||||
abortIfNeeded(signal);
|
||||
const length = Math.min(
|
||||
requiredLength() - available,
|
||||
fileSize - offset - available,
|
||||
);
|
||||
const chunk = Buffer.alloc(length);
|
||||
const read = await handle.read(chunk, 0, length, offset + available);
|
||||
abortIfNeeded(signal);
|
||||
if (read.bytesRead === 0) break;
|
||||
chunk.copy(window, available, 0, read.bytesRead);
|
||||
available += read.bytesRead;
|
||||
}
|
||||
if (available === 0) return undefined;
|
||||
return { offset, bytes: window.subarray(0, available) };
|
||||
};
|
||||
|
||||
const hasKolyTrailer = async (
|
||||
|
||||
@@ -344,13 +344,17 @@ const digestArtifactEntry = async (
|
||||
}
|
||||
const classified =
|
||||
entry.kind === "slice"
|
||||
? ({ kind: "universal-slice", format: "mach-o" } as const)
|
||||
? {
|
||||
kind: "universal-slice" as const,
|
||||
format: "mach-o" as const,
|
||||
contentFormat: classifyArtifactBytes(digest, digest.bytes),
|
||||
}
|
||||
: classifyArtifactContent(logicalPath, digest, digest.bytes);
|
||||
return {
|
||||
node: createArtifactNode({
|
||||
sha256: digest.sha256,
|
||||
size: digest.bytes,
|
||||
format: classifyArtifactBytes(digest.prefix, digest.bytes),
|
||||
format: classified.contentFormat,
|
||||
contentState: "embedded",
|
||||
}),
|
||||
mismatched,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { deflateSync } from "node:zlib";
|
||||
import { pe } from "../../domain/binaryTarget.fixture.js";
|
||||
|
||||
import { PYINSTALLER_COOKIE_MAGIC } from "./CArchiveLayout.js";
|
||||
|
||||
@@ -28,8 +29,10 @@ export interface CArchiveFixtureMember {
|
||||
readonly declaredUncompressedSize?: number;
|
||||
}
|
||||
|
||||
/** Build a tiny ELF with an appended CArchive, optionally followed by a trailer. */
|
||||
/** Build a tiny native carrier with an appended CArchive and optional trailer. */
|
||||
export const cArchiveExecutable = (input: {
|
||||
readonly carrier?: "elf" | "pe";
|
||||
readonly peHeaderOffset?: number;
|
||||
readonly layout: "2.0" | "2.1";
|
||||
readonly pythonVersion: number;
|
||||
readonly pythonLibrary?: string;
|
||||
@@ -70,7 +73,9 @@ export const cArchiveExecutable = (input: {
|
||||
tocLength: toc.length,
|
||||
});
|
||||
return Buffer.concat([
|
||||
ELF_PREFIX,
|
||||
input.carrier === "pe"
|
||||
? pe(0x8664, input.peHeaderOffset ?? 64)
|
||||
: ELF_PREFIX,
|
||||
...payloads,
|
||||
toc,
|
||||
cookie,
|
||||
|
||||
@@ -15,6 +15,8 @@ import type { PyInstallerEntryMetadata } from "../../domain/artifactFormatMetada
|
||||
|
||||
const CRYPTO_KEY_NAME = "pyimod00_crypto_key";
|
||||
|
||||
export type CarrierPathDialect = "posix" | "windows" | "unknown";
|
||||
|
||||
/** Parsed facts and the descriptor identity used to locate a retained image. */
|
||||
export interface PyzImage {
|
||||
readonly archive: PyzArchive;
|
||||
@@ -59,6 +61,7 @@ export const projectStoredMember = (
|
||||
entry: CArchiveEntry,
|
||||
archiveStart: number,
|
||||
pythonMagic: Buffer | undefined,
|
||||
pathDialect: CarrierPathDialect,
|
||||
): PyInstallerMember => {
|
||||
if (isCArchiveDeclaration(entry))
|
||||
return {
|
||||
@@ -83,7 +86,7 @@ export const projectStoredMember = (
|
||||
pythonMagic !== undefined && isBytecodeType(entry.typeCode)
|
||||
? reconstructedPycHeader(pythonMagic)
|
||||
: undefined;
|
||||
const path = canonicalMemberPath(entry.name);
|
||||
const path = canonicalMemberPath(entry.name, pathDialect);
|
||||
const symlink = entry.typeCode === "n";
|
||||
const limitations: string[] = [];
|
||||
if (symlink) limitations.push("PyInstaller symlink target was not read.");
|
||||
@@ -116,9 +119,10 @@ export const projectPyzContainer = (
|
||||
archiveStart: number,
|
||||
image: PyzImage,
|
||||
encrypted: boolean,
|
||||
pathDialect: CarrierPathDialect,
|
||||
): PyInstallerMember => {
|
||||
const { archive } = image;
|
||||
const path = canonicalMemberPath(entry.name);
|
||||
const path = canonicalMemberPath(entry.name, pathDialect);
|
||||
const byteOffset = sourceByteOffset(entry, archiveStart);
|
||||
const limitations: string[] = [];
|
||||
if (archive.kind === "opaque")
|
||||
@@ -198,6 +202,7 @@ export const projectPyzMember = (
|
||||
item: PyzEntry,
|
||||
encryptionMarker: boolean,
|
||||
pythonMagic: Buffer | undefined,
|
||||
pathDialect: CarrierPathDialect,
|
||||
range:
|
||||
| {
|
||||
readonly handle: FileHandle;
|
||||
@@ -206,7 +211,11 @@ export const projectPyzMember = (
|
||||
}
|
||||
| undefined,
|
||||
): PyInstallerMember => {
|
||||
const path = pyzMemberPath(canonicalMemberPath(container.name), item);
|
||||
const path = pyzMemberPath(
|
||||
canonicalMemberPath(container.name, pathDialect),
|
||||
item,
|
||||
pathDialect,
|
||||
);
|
||||
const encrypted = pyzEntryEncrypted(item, encryptionMarker);
|
||||
if (item.namespace)
|
||||
return {
|
||||
@@ -282,14 +291,17 @@ export const projectPyzMember = (
|
||||
};
|
||||
|
||||
/**
|
||||
* CArchive and PYZ data names treat `\` as a separator. The logical path is
|
||||
* CArchive and PYZ data names use the carrier's path dialect. The logical path is
|
||||
* canonical; the stored spelling is kept when that conversion changes it.
|
||||
* `source_location` is an offset plus declared size, so a compressed member
|
||||
* publishes a file span only when that size is the physical extent. Otherwise
|
||||
* the span stays unknown, as ZIP does for compressed entries.
|
||||
*/
|
||||
const canonicalMemberPath = (storedName: string): string =>
|
||||
storedName.replaceAll("\\", "/");
|
||||
const canonicalMemberPath = (
|
||||
storedName: string,
|
||||
pathDialect: CarrierPathDialect,
|
||||
): string =>
|
||||
pathDialect === "windows" ? storedName.replaceAll("\\", "/") : storedName;
|
||||
|
||||
const sourceByteOffset = (
|
||||
entry: CArchiveEntry,
|
||||
@@ -297,9 +309,13 @@ const sourceByteOffset = (
|
||||
): number | null =>
|
||||
entry.compressionFlag === 0 ? archiveStart + entry.offset : null;
|
||||
|
||||
const pyzMemberPath = (container: string, item: PyzEntry): string => {
|
||||
const pyzMemberPath = (
|
||||
container: string,
|
||||
item: PyzEntry,
|
||||
pathDialect: CarrierPathDialect,
|
||||
): string => {
|
||||
if (item.typeCode === 2)
|
||||
return `${container}/${canonicalMemberPath(item.name)}`;
|
||||
return `${container}/${canonicalMemberPath(item.name, pathDialect)}`;
|
||||
const dotted = item.name.replaceAll(".", "/");
|
||||
if (item.namespace) return `${container}/${dotted}`;
|
||||
if (item.package) return `${container}/${dotted}/__init__.pyc`;
|
||||
|
||||
@@ -849,6 +849,7 @@ it("inventories and extracts a Windows member name under its canonical path", as
|
||||
const path = await writeArchive(
|
||||
"windows",
|
||||
cArchiveExecutable({
|
||||
carrier: "pe",
|
||||
layout: "2.1",
|
||||
pythonVersion: 309,
|
||||
pythonLibrary: "python39.dll",
|
||||
@@ -880,7 +881,7 @@ it("inventories and extracts a Windows member name under its canonical path", as
|
||||
await materializeArtifactInventory(
|
||||
{
|
||||
inputPath: path,
|
||||
inputFormat: "elf",
|
||||
inputFormat: "pe",
|
||||
outputRoot: output,
|
||||
environment: process.env,
|
||||
integrityPolicy: "fail",
|
||||
@@ -894,6 +895,92 @@ it("inventories and extracts a Windows member name under its canonical path", as
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves distinct POSIX backslash and slash member paths", async () => {
|
||||
const backslashName = String.raw`a\b.txt`;
|
||||
const slashName = "a/b.txt";
|
||||
const backslashPayload = Buffer.from("literal-backslash");
|
||||
const slashPayload = Buffer.from("nested-slash");
|
||||
const path = await writeArchive(
|
||||
"posix-paths",
|
||||
cArchiveExecutable({
|
||||
layout: "2.1",
|
||||
pythonVersion: 309,
|
||||
pythonLibrary: "python39.dll",
|
||||
members: [
|
||||
{
|
||||
name: backslashName,
|
||||
typeCode: "x",
|
||||
payload: backslashPayload,
|
||||
compress: false,
|
||||
},
|
||||
{
|
||||
name: slashName,
|
||||
typeCode: "x",
|
||||
payload: slashPayload,
|
||||
compress: false,
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const snapshot = await scanCanonicalArtifactInventory(path);
|
||||
const backslashEntry = snapshot.occurrences.find(
|
||||
(occurrence) => occurrence.logical_path === backslashName,
|
||||
);
|
||||
const slashEntry = snapshot.occurrences.find(
|
||||
(occurrence) => occurrence.logical_path === slashName,
|
||||
);
|
||||
expect(backslashEntry?.format_metadata).toMatchObject({
|
||||
source_name: backslashName,
|
||||
});
|
||||
expect(slashEntry?.format_metadata).toMatchObject({ source_name: slashName });
|
||||
expect(snapshot.coverage.status).toBe("complete");
|
||||
if (process.platform === "win32") return;
|
||||
|
||||
const directory = await createTestTempDirectory("rea-pyinstaller-posix-out-");
|
||||
const output = join(directory, "extracted");
|
||||
await materializeArtifactInventory(
|
||||
{
|
||||
inputPath: path,
|
||||
inputFormat: "elf",
|
||||
outputRoot: output,
|
||||
environment: process.env,
|
||||
integrityPolicy: "fail",
|
||||
},
|
||||
path,
|
||||
snapshot,
|
||||
);
|
||||
expect(await readFile(join(output, backslashName))).toEqual(backslashPayload);
|
||||
expect(await readFile(join(output, "a", "b.txt"))).toEqual(slashPayload);
|
||||
});
|
||||
|
||||
it("reads a PE signature beyond the carrier prefix before projecting paths", async () => {
|
||||
const path = await writeArchive(
|
||||
"distant-pe",
|
||||
cArchiveExecutable({
|
||||
carrier: "pe",
|
||||
peHeaderOffset: 8190,
|
||||
layout: "2.1",
|
||||
pythonVersion: 309,
|
||||
members: [
|
||||
{
|
||||
name: String.raw`pkg\data.bin`,
|
||||
typeCode: "x",
|
||||
payload: Buffer.from("windows-path"),
|
||||
compress: false,
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const reader = new PyInstallerArtifactReader(path);
|
||||
try {
|
||||
const entries = [];
|
||||
for await (const entry of reader.entries()) entries.push(entry);
|
||||
expect(entries.map((entry) => entry.path)).toContain("pkg/data.bin");
|
||||
} finally {
|
||||
await reader.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects a stored member whose declared size disagrees with its bytes", async () => {
|
||||
const path = await writeArchive(
|
||||
"stored-size",
|
||||
|
||||
@@ -127,6 +127,7 @@ it.each([true, false])(
|
||||
};
|
||||
await withPyInstallerReader(
|
||||
cArchiveExecutable({
|
||||
carrier: "pe",
|
||||
layout: "2.1",
|
||||
pythonVersion: 309,
|
||||
members: [
|
||||
|
||||
@@ -24,6 +24,7 @@ import {
|
||||
} from "../mobileApplicationGraphProjection.js";
|
||||
import { compareUnicodeCodePoints } from "../unicodeCodePointOrder.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { isV1SignatureMetadataName } from "../apkSignature/apkV1MetadataName.js";
|
||||
|
||||
/** Deterministic, execution-free Android application inventory projection. */
|
||||
export const androidApplicationProjectionResultSchema = z.strictObject({
|
||||
@@ -173,9 +174,7 @@ const classify = (all: readonly Component[]) => ({
|
||||
format === "elf" || /(?:^|\/)lib\/[^/]+\/[^/]+\.so$/iu.test(path),
|
||||
),
|
||||
javascript: all.filter(({ format }) => format === "javascript-bundle"),
|
||||
signing: all.filter(({ path }) =>
|
||||
/^META-INF\/[^/]+\.(?:MF|RSA|DSA|EC|SF)$/iu.test(path),
|
||||
),
|
||||
signing: all.filter(({ path }) => isV1SignatureMetadataName(path)),
|
||||
});
|
||||
|
||||
const runtimeFamilies = (
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
/** Exact path-name rules used to select APK v1 signing metadata. */
|
||||
export const APK_V1_META_INF_PREFIX = "META-INF/";
|
||||
export const APK_V1_MANIFEST_PATH = `${APK_V1_META_INF_PREFIX}MANIFEST.MF`;
|
||||
|
||||
const META_INF_PREFIX_BYTES = Uint8Array.from(
|
||||
[...APK_V1_META_INF_PREFIX].map((character) => character.charCodeAt(0)),
|
||||
);
|
||||
const BLOCK_SUFFIXES = [".RSA", ".DSA", ".EC"] as const;
|
||||
|
||||
export const isV1SignatureFileName = (name: string): boolean =>
|
||||
name.endsWith(".SF");
|
||||
|
||||
export const isV1SignatureBlockName = (name: string): boolean =>
|
||||
BLOCK_SUFFIXES.some((suffix) => name.endsWith(suffix));
|
||||
|
||||
export const v1SignatureFileNameForBlock = (block: string): string =>
|
||||
`${block.slice(0, block.lastIndexOf("."))}.SF`;
|
||||
|
||||
export const isV1SignatureMetadataName = (name: string): boolean =>
|
||||
name.startsWith(APK_V1_META_INF_PREFIX) &&
|
||||
(name === APK_V1_MANIFEST_PATH ||
|
||||
isV1SignatureFileName(name) ||
|
||||
isV1SignatureBlockName(name));
|
||||
|
||||
/** Test the raw central-directory name before decoding noncandidate ZIP paths. */
|
||||
export const startsWithV1MetaInfPrefix = (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
length: number,
|
||||
): boolean => {
|
||||
if (length < META_INF_PREFIX_BYTES.length) return false;
|
||||
for (let index = 0; index < META_INF_PREFIX_BYTES.length; index += 1)
|
||||
if (bytes[start + index] !== META_INF_PREFIX_BYTES[index]) return false;
|
||||
return true;
|
||||
};
|
||||
@@ -25,6 +25,15 @@ import {
|
||||
sameZipName,
|
||||
type StoredZipName,
|
||||
} from "./zipEntryName.js";
|
||||
import {
|
||||
APK_V1_MANIFEST_PATH,
|
||||
APK_V1_META_INF_PREFIX,
|
||||
isV1SignatureBlockName,
|
||||
isV1SignatureFileName,
|
||||
isV1SignatureMetadataName,
|
||||
startsWithV1MetaInfPrefix,
|
||||
v1SignatureFileNameForBlock,
|
||||
} from "./apkV1MetadataName.js";
|
||||
|
||||
const CD_SIGNATURE = 0x02014b50;
|
||||
const CD_FIXED_SIZE = 46;
|
||||
@@ -56,37 +65,6 @@ export const MAX_V1_SIGNERS = 32;
|
||||
/** Duplicated central-directory names decoded for the report; the rest are counted. */
|
||||
export const MAX_DUPLICATE_NAME_SAMPLE = 5;
|
||||
|
||||
const META_INF = "META-INF/";
|
||||
const MANIFEST = "META-INF/MANIFEST.MF";
|
||||
const META_INF_BYTES = Uint8Array.from(
|
||||
[...META_INF].map((c) => c.charCodeAt(0)),
|
||||
);
|
||||
|
||||
/** True when the raw name starts with `META-INF/` (apksig's selection). */
|
||||
const startsWithMetaInf = (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
length: number,
|
||||
): boolean => {
|
||||
if (length < META_INF_BYTES.length) return false;
|
||||
for (let i = 0; i < META_INF_BYTES.length; i += 1)
|
||||
if (bytes[start + i] !== META_INF_BYTES[i]) return false;
|
||||
return true;
|
||||
};
|
||||
|
||||
const BLOCK_SUFFIXES = [".RSA", ".DSA", ".EC"] as const;
|
||||
const isSignatureFile = (name: string): boolean => name.endsWith(".SF");
|
||||
const isSignatureBlock = (name: string): boolean =>
|
||||
BLOCK_SUFFIXES.some((suffix) => name.endsWith(suffix));
|
||||
|
||||
/** The `.SF` file apksig pairs with a signature block. */
|
||||
const signatureFileFor = (block: string): string =>
|
||||
`${block.slice(0, block.lastIndexOf("."))}.SF`;
|
||||
|
||||
/** The one owner of which `META-INF/` names take part in v1 signing. */
|
||||
const isV1SignatureName = (name: string): boolean =>
|
||||
name === MANIFEST || isSignatureFile(name) || isSignatureBlock(name);
|
||||
|
||||
/** A signing-relevant central-directory entry and where its data is stored. */
|
||||
export interface ApkV1Entry {
|
||||
readonly name: string;
|
||||
@@ -331,10 +309,10 @@ export const centralDirectoryV1SignatureEntries = (
|
||||
});
|
||||
starts[i] = nameStart;
|
||||
lengths[i] = nameLength;
|
||||
if (startsWithMetaInf(bytes, nameStart, nameLength)) {
|
||||
if (startsWithV1MetaInfPrefix(bytes, nameStart, nameLength)) {
|
||||
const rawName = bytes.subarray(nameStart, nameStart + nameLength);
|
||||
const name = decodeZipName(rawName);
|
||||
if (isV1SignatureName(name)) {
|
||||
if (isV1SignatureMetadataName(name)) {
|
||||
if (entries.length >= MAX_V1_SIGNATURE_ENTRIES)
|
||||
return err({
|
||||
kind: "budget-exceeded",
|
||||
@@ -373,7 +351,9 @@ export const centralDirectoryV1SignatureEntries = (
|
||||
export const detectV1Signature = (
|
||||
entries: readonly ApkV1Entry[],
|
||||
): ApkV1Signature => {
|
||||
const selected = entries.filter((entry) => entry.name.startsWith(META_INF));
|
||||
const selected = entries.filter((entry) =>
|
||||
entry.name.startsWith(APK_V1_META_INF_PREFIX),
|
||||
);
|
||||
// Case-sensitive throughout, as apksig matches JAR signature entries.
|
||||
const seen = new Set<string>();
|
||||
const duplicates = new Set<string>();
|
||||
@@ -382,24 +362,24 @@ export const detectV1Signature = (
|
||||
else seen.add(name);
|
||||
const signatureFiles = selected
|
||||
.map((entry) => entry.name)
|
||||
.filter(isSignatureFile);
|
||||
const blocks = selected.filter((entry) => isSignatureBlock(entry.name));
|
||||
.filter(isV1SignatureFileName);
|
||||
const blocks = selected.filter((entry) => isV1SignatureBlockName(entry.name));
|
||||
const sfNames = new Set(signatureFiles);
|
||||
const paired = new Set<string>();
|
||||
const signers = blocks.flatMap((block) => {
|
||||
const signatureFile = signatureFileFor(block.name);
|
||||
const signatureFile = v1SignatureFileNameForBlock(block.name);
|
||||
if (!sfNames.has(signatureFile) || paired.has(block.name)) return [];
|
||||
paired.add(block.name);
|
||||
// apksig reads every signer's manifest, so a duplicate one affects them all.
|
||||
const duplicate =
|
||||
[block.name, signatureFile, MANIFEST].find((name) =>
|
||||
[block.name, signatureFile, APK_V1_MANIFEST_PATH].find((name) =>
|
||||
duplicates.has(name),
|
||||
) ?? null;
|
||||
return [{ signatureFile, block, duplicate }];
|
||||
});
|
||||
return {
|
||||
present: signers.length > 0,
|
||||
hasManifest: selected.some((entry) => entry.name === MANIFEST),
|
||||
hasManifest: selected.some((entry) => entry.name === APK_V1_MANIFEST_PATH),
|
||||
signatureFiles,
|
||||
blockFiles: blocks.map((block) => block.name),
|
||||
signers,
|
||||
|
||||
@@ -10,7 +10,10 @@ import type {
|
||||
} from "./binaryTargetTypes.js";
|
||||
import { err, ok, type Result } from "./result.js";
|
||||
import { readElfFixedHeader } from "./elfFixedHeader.js";
|
||||
import { parseMzHeaderEvidence } from "./dosMz.js";
|
||||
import {
|
||||
parseMzHeaderEvidence,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "./dosMz.js";
|
||||
import {
|
||||
decodePeMachine,
|
||||
peOptionalHeaderMagic,
|
||||
@@ -106,7 +109,7 @@ export const parseExecutableHeader = (
|
||||
bytes: Buffer,
|
||||
hostArchitecture: NodeJS.Architecture,
|
||||
fileSize = bytes.length,
|
||||
mzWindowsHeaderSignature?: Buffer,
|
||||
mzWindowsHeaderSignature?: MzWindowsHeaderSignatureEvidence,
|
||||
fatEvidence?: FatSliceHeaderEvidence,
|
||||
): Result<ExecutableMetadata, string> => {
|
||||
if (
|
||||
@@ -436,7 +439,7 @@ const parseElf = (
|
||||
const parseMz = (
|
||||
bytes: Buffer,
|
||||
fileSize: number,
|
||||
signatureAtOffset?: Buffer,
|
||||
signatureAtOffset?: MzWindowsHeaderSignatureEvidence,
|
||||
): Result<ExecutableMetadata, string> => {
|
||||
const evidence = parseMzHeaderEvidence(bytes, fileSize, signatureAtOffset);
|
||||
const { windowsHeader } = evidence;
|
||||
|
||||
@@ -264,11 +264,10 @@ describe("MZ Windows header observations", () => {
|
||||
offset,
|
||||
});
|
||||
expect(
|
||||
observeMzWindowsHeader(
|
||||
prefix,
|
||||
offset + 512,
|
||||
Buffer.from("PE\0\0", "binary"),
|
||||
),
|
||||
observeMzWindowsHeader(prefix, offset + 512, {
|
||||
offset,
|
||||
bytes: Buffer.from("PE\0\0", "binary"),
|
||||
}),
|
||||
).toEqual({ state: "declared", offset, signature: "PE" });
|
||||
});
|
||||
|
||||
|
||||
+14
-5
@@ -39,6 +39,12 @@ export type MzWindowsHeaderObservation =
|
||||
readonly signature: "PE" | "NE" | "LE" | "LX";
|
||||
};
|
||||
|
||||
/** Exact signature bytes observed at the DOS header's declared offset. */
|
||||
export interface MzWindowsHeaderSignatureEvidence {
|
||||
readonly offset: number;
|
||||
readonly bytes: Buffer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Observe the Windows new-header pointer without mistaking short DOS module
|
||||
* bytes for e_lfanew. A nonzero pointer in a complete DOS header is only a
|
||||
@@ -51,7 +57,7 @@ export type MzWindowsHeaderObservation =
|
||||
export const observeMzWindowsHeader = (
|
||||
bytes: Buffer,
|
||||
fileSize = bytes.length,
|
||||
signatureAtOffset?: Buffer,
|
||||
signatureEvidence?: MzWindowsHeaderSignatureEvidence,
|
||||
): MzWindowsHeaderObservation => {
|
||||
if (bytes.length < 64)
|
||||
return fileSize >= 64 ? { state: "header-needed" } : { state: "absent" };
|
||||
@@ -71,7 +77,10 @@ export const observeMzWindowsHeader = (
|
||||
offset < bytes.length
|
||||
? bytes.subarray(offset, Math.min(offset + 4, bytes.length))
|
||||
: undefined;
|
||||
const available = signatureAtOffset ?? inlineSignature;
|
||||
const available =
|
||||
signatureEvidence?.offset === offset
|
||||
? signatureEvidence.bytes
|
||||
: inlineSignature;
|
||||
const signature = windowsHeaderSignature(available);
|
||||
if (signature !== "other" && signature !== "unobserved")
|
||||
return { state: "declared", offset, signature };
|
||||
@@ -85,7 +94,7 @@ export const observeMzWindowsHeader = (
|
||||
available.toString("latin1", 0, 2) === "PE" &&
|
||||
available.length < 4 &&
|
||||
remaining >= 4);
|
||||
if (needsMore && signatureAtOffset === undefined)
|
||||
if (needsMore && signatureEvidence?.offset !== offset)
|
||||
return { state: "signature-needed", offset };
|
||||
return { state: "candidate", offset, signature };
|
||||
}
|
||||
@@ -177,10 +186,10 @@ export interface MzHeaderEvidence {
|
||||
export const parseMzHeaderEvidence = (
|
||||
bytes: Buffer,
|
||||
fileSize = bytes.length,
|
||||
signatureAtOffset?: Buffer,
|
||||
signatureEvidence?: MzWindowsHeaderSignatureEvidence,
|
||||
): MzHeaderEvidence => ({
|
||||
dos: parseDosMzHeader(bytes, fileSize),
|
||||
windowsHeader: observeMzWindowsHeader(bytes, fileSize, signatureAtOffset),
|
||||
windowsHeader: observeMzWindowsHeader(bytes, fileSize, signatureEvidence),
|
||||
});
|
||||
|
||||
const validateRelocations = (
|
||||
|
||||
@@ -9,6 +9,11 @@ export const SQLITE_PROVIDER_IDENTITY = {
|
||||
version: null,
|
||||
} as const;
|
||||
|
||||
/** SQLite exposes this table as a row source but omits its own catalog entry. */
|
||||
export const SQLITE_SCHEMA_TABLE_NAME = "sqlite_schema";
|
||||
export const sqliteSchemaHasObjectDeclaration = (name: string): boolean =>
|
||||
name !== SQLITE_SCHEMA_TABLE_NAME;
|
||||
|
||||
/** Default ordinary-table sample size when the caller selects a table. */
|
||||
export const SQLITE_ROW_LIMIT_DEFAULT = 20;
|
||||
/** Execution budget, bounded by the runtime's signed 32-bit timer range. */
|
||||
|
||||
@@ -58,15 +58,17 @@ export const attestGhidraDosSnapshot = async (
|
||||
}
|
||||
if (captured !== captureBytes) throw new RegularFileChangedError(path);
|
||||
const windowsHeader = observeMzWindowsHeader(prefix, fileBytes);
|
||||
let signature: Buffer | undefined;
|
||||
let signature:
|
||||
| { readonly offset: number; readonly bytes: Buffer }
|
||||
| undefined;
|
||||
if (windowsHeader.state === "signature-needed") {
|
||||
const size = Math.min(4, fileBytes - windowsHeader.offset);
|
||||
signature = Buffer.allocUnsafe(size);
|
||||
const signatureBytes = Buffer.allocUnsafe(size);
|
||||
let read = 0;
|
||||
while (read < size) {
|
||||
if (signal?.aborted) throw cancelled(signal.reason);
|
||||
const { bytesRead } = await handle.read(
|
||||
signature,
|
||||
signatureBytes,
|
||||
read,
|
||||
size - read,
|
||||
windowsHeader.offset + read,
|
||||
@@ -74,6 +76,10 @@ export const attestGhidraDosSnapshot = async (
|
||||
if (bytesRead === 0) throw new RegularFileChangedError(path);
|
||||
read += bytesRead;
|
||||
}
|
||||
signature = {
|
||||
offset: windowsHeader.offset,
|
||||
bytes: signatureBytes,
|
||||
};
|
||||
}
|
||||
const prepared = prepareGhidraDosLoadImage(prefix, fileBytes, signature);
|
||||
if (!prepared.ok) return prepared;
|
||||
|
||||
@@ -5,7 +5,11 @@ import {
|
||||
type NativeLoadImageCheck,
|
||||
type NativeLoadImageObservation,
|
||||
} from "../domain/native/nativeLoadImage.js";
|
||||
import { parseMzHeaderEvidence, type DosMzHeader } from "../domain/dosMz.js";
|
||||
import {
|
||||
parseMzHeaderEvidence,
|
||||
type DosMzHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "../domain/dosMz.js";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
import type { BinaryTarget } from "../domain/binaryTargetTypes.js";
|
||||
import { validateDosComLength } from "../domain/dosCom.js";
|
||||
@@ -55,7 +59,7 @@ export type GhidraDosImage = {
|
||||
export const prepareGhidraDosLoadImage = (
|
||||
bytes: Buffer,
|
||||
fileBytes: number,
|
||||
signatureAtOffset?: Buffer,
|
||||
signatureAtOffset?: MzWindowsHeaderSignatureEvidence,
|
||||
): Result<GhidraDosImage, string> => {
|
||||
const evidence = parseMzHeaderEvidence(bytes, fileBytes, signatureAtOffset);
|
||||
if (evidence.windowsHeader.state === "candidate")
|
||||
|
||||
@@ -3,16 +3,21 @@ import { readFileSync } from "node:fs";
|
||||
import { expect, it } from "vitest";
|
||||
import { inspectElfNotesInputSchema } from "../../domain/native/elfNotes.js";
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
import { BinaryFormatFailure } from "../metadata/BinaryContainer.js";
|
||||
import {
|
||||
compressedElfNoteFixture,
|
||||
elfNote,
|
||||
elfNoteFixture,
|
||||
elfNoteTablePastEnd,
|
||||
elfExtendedSectionCountFixture,
|
||||
gnuAbiTagNote,
|
||||
gnuBuildIdNote,
|
||||
gnuPropertyNote,
|
||||
} from "./ElfNotes.fixture.js";
|
||||
import { parseElfNotes } from "./ElfNoteParser.js";
|
||||
import {
|
||||
ElfNoteSectionNameBudgetFailure,
|
||||
parseElfNotes,
|
||||
} from "./ElfNoteParser.js";
|
||||
|
||||
const defaults = inspectElfNotesInputSchema.parse({
|
||||
path: "/artifacts/app",
|
||||
@@ -133,14 +138,51 @@ it("keeps note facts but marks section names unavailable for unknown compression
|
||||
);
|
||||
});
|
||||
|
||||
it("bounds compressed section-name expansion with the source bytes", () => {
|
||||
it("tags valid extended section numbering as unsupported", () => {
|
||||
let failure: unknown;
|
||||
try {
|
||||
parse(elfExtendedSectionCountFixture());
|
||||
} catch (cause) {
|
||||
failure = cause;
|
||||
}
|
||||
expect(failure).toBeInstanceOf(BinaryFormatFailure);
|
||||
expect(failure).toMatchObject({
|
||||
kind: "unsupported",
|
||||
message: "ELF extended numbering is unsupported.",
|
||||
});
|
||||
});
|
||||
|
||||
it("admits compressed section names when max_file_bytes includes expansion", () => {
|
||||
const bytes = compressedElfNoteFixture();
|
||||
const sectionOffset = Number(bytes.readBigUInt64LE(40));
|
||||
const nameHeader = sectionOffset + bytes.readUInt16LE(62) * 64;
|
||||
const namesOffset = Number(bytes.readBigUInt64LE(nameHeader + 24));
|
||||
const decodedNamesBytes = Number(bytes.readBigUInt64LE(namesOffset + 8));
|
||||
const minimumMaxFileBytes = bytes.length + decodedNamesBytes;
|
||||
const input = inspectElfNotesInputSchema.parse({
|
||||
path: defaults.path,
|
||||
max_file_bytes: bytes.length + 1,
|
||||
});
|
||||
expect(() => parse(bytes, input)).toThrow(ArtifactReaderFailure);
|
||||
expect(() => parse(bytes, input)).toThrow(/memory budget/u);
|
||||
let failure: unknown;
|
||||
try {
|
||||
parse(bytes, input);
|
||||
} catch (cause) {
|
||||
failure = cause;
|
||||
}
|
||||
expect(failure).toBeInstanceOf(ElfNoteSectionNameBudgetFailure);
|
||||
expect(failure).toMatchObject({
|
||||
minimumMaxFileBytes: BigInt(minimumMaxFileBytes),
|
||||
selectedMaxFileBytes: bytes.length + 1,
|
||||
});
|
||||
|
||||
const admitted = parse(
|
||||
bytes,
|
||||
inspectElfNotesInputSchema.parse({
|
||||
path: defaults.path,
|
||||
max_file_bytes: minimumMaxFileBytes,
|
||||
}),
|
||||
);
|
||||
expect(admitted.tables[0]?.name).toBe(".note.gnu.build-id");
|
||||
});
|
||||
|
||||
it("reports independent PT_NOTE and SHT_NOTE declarations of the same bytes", () => {
|
||||
@@ -327,7 +369,18 @@ const unterminatedNoteSections = (
|
||||
|
||||
it("bounds repeated scans of an unterminated section name", () => {
|
||||
expect(parse(unterminatedNoteSections(1, 64)).tables[0]?.name).toBeNull();
|
||||
expect(() => parse(unterminatedNoteSections(3, 400))).toThrow(/byte budget/u);
|
||||
let failure: unknown;
|
||||
try {
|
||||
parse(unterminatedNoteSections(3, 400));
|
||||
} catch (cause) {
|
||||
failure = cause;
|
||||
}
|
||||
expect(failure).toBeInstanceOf(ArtifactReaderFailure);
|
||||
expect(failure).toMatchObject({ reason: "limit", boundary: "memory" });
|
||||
expect(failure).toHaveProperty(
|
||||
"message",
|
||||
expect.stringMatching(/byte budget/u),
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects note text that repeats beyond the shared byte budget", () => {
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
elfObjectTypeName,
|
||||
gnuNoteTypeName,
|
||||
} from "../../domain/native/elfNotes.js";
|
||||
import { BinaryFormatFailure } from "../metadata/BinaryContainer.js";
|
||||
import { decodeElfSectionNameTable } from "../metadata/ElfSectionNameTable.js";
|
||||
|
||||
const PT_NOTE = 4;
|
||||
@@ -29,6 +30,21 @@ interface TextBudget {
|
||||
readonly scans: Map<string, number>;
|
||||
}
|
||||
|
||||
/** Caller can raise max_file_bytes to admit the decoded section-name table. */
|
||||
export class ElfNoteSectionNameBudgetFailure extends Error {
|
||||
readonly _tag = "ElfNoteSectionNameBudgetFailure";
|
||||
|
||||
constructor(
|
||||
readonly minimumMaxFileBytes: bigint,
|
||||
readonly selectedMaxFileBytes: number,
|
||||
) {
|
||||
super(
|
||||
`ELF source and decoded section names require ${String(minimumMaxFileBytes)} bytes, above the selected max_file_bytes (${String(selectedMaxFileBytes)}).`,
|
||||
);
|
||||
this.name = "ElfNoteSectionNameBudgetFailure";
|
||||
}
|
||||
}
|
||||
|
||||
const fail = (message: string): never => {
|
||||
throw new ArtifactReaderFailure("format", message);
|
||||
};
|
||||
@@ -304,7 +320,10 @@ export const parseElfNotes = (
|
||||
shstrndx === 0xffff ||
|
||||
(shoff !== 0 && shnum === 0)
|
||||
)
|
||||
return fail("ELF extended numbering is unsupported.");
|
||||
throw new BinaryFormatFailure(
|
||||
"unsupported",
|
||||
"ELF extended numbering is unsupported.",
|
||||
);
|
||||
if (phnum !== 0) {
|
||||
if (phentsize < (wide ? 56 : 32))
|
||||
return fail("ELF program header width is invalid.");
|
||||
@@ -344,10 +363,9 @@ export const parseElfNotes = (
|
||||
);
|
||||
if (decoded.state === "malformed") return fail(decoded.message);
|
||||
if (decoded.state === "limit")
|
||||
throw new ArtifactReaderFailure(
|
||||
"limit",
|
||||
`ELF source and compressed section names exceed the ${String(input.max_file_bytes)}-byte memory budget.`,
|
||||
{ boundary: "memory" },
|
||||
throw new ElfNoteSectionNameBudgetFailure(
|
||||
BigInt(bytes.length) + decoded.requiredBytes,
|
||||
input.max_file_bytes,
|
||||
);
|
||||
if (decoded.state === "unsupported") {
|
||||
shstrtab = null;
|
||||
|
||||
@@ -209,6 +209,46 @@ export const elfNoteFixture = (options: ElfNoteImageOptions = {}): Buffer => {
|
||||
return bytes;
|
||||
};
|
||||
|
||||
/** Valid ELF64 section count stored in section header zero per the gABI. */
|
||||
export const elfExtendedSectionCountFixture = (): Buffer => {
|
||||
const count = 0xff00;
|
||||
const sectionOffset = 0x200;
|
||||
const noteOffset = 0x100;
|
||||
const namesOffset = 0x180;
|
||||
const note = gnuBuildIdNote("12345678");
|
||||
const names = Buffer.from("\0.note.gnu.build-id\0.shstrtab\0", "latin1");
|
||||
const bytes = Buffer.alloc(sectionOffset + count * 64);
|
||||
bytes.set([0x7f, 0x45, 0x4c, 0x46, 2, 1, 1]);
|
||||
bytes.writeUInt16LE(1, 16);
|
||||
bytes.writeUInt16LE(62, 18);
|
||||
bytes.writeUInt32LE(1, 20);
|
||||
bytes.writeBigUInt64LE(BigInt(sectionOffset), 40);
|
||||
bytes.writeUInt16LE(64, 52);
|
||||
bytes.writeUInt16LE(56, 54);
|
||||
bytes.writeUInt16LE(64, 58);
|
||||
bytes.writeUInt16LE(0, 60);
|
||||
bytes.writeUInt16LE(2, 62);
|
||||
bytes.writeBigUInt64LE(BigInt(count), sectionOffset + 32);
|
||||
|
||||
const noteSection = sectionOffset + 64;
|
||||
bytes.writeUInt32LE(1, noteSection);
|
||||
bytes.writeUInt32LE(7, noteSection + 4);
|
||||
bytes.writeBigUInt64LE(BigInt(noteOffset), noteSection + 24);
|
||||
bytes.writeBigUInt64LE(BigInt(note.length), noteSection + 32);
|
||||
bytes.writeBigUInt64LE(4n, noteSection + 48);
|
||||
|
||||
const namesSection = sectionOffset + 2 * 64;
|
||||
bytes.writeUInt32LE(20, namesSection);
|
||||
bytes.writeUInt32LE(3, namesSection + 4);
|
||||
bytes.writeBigUInt64LE(BigInt(namesOffset), namesSection + 24);
|
||||
bytes.writeBigUInt64LE(BigInt(names.length), namesSection + 32);
|
||||
bytes.writeBigUInt64LE(1n, namesSection + 48);
|
||||
|
||||
note.copy(bytes, noteOffset);
|
||||
names.copy(bytes, namesOffset);
|
||||
return bytes;
|
||||
};
|
||||
|
||||
/** ELF note fixture with a zlib-compressed section-name string table. */
|
||||
export const compressedElfNoteFixture = (
|
||||
bits: 32 | 64 = 64,
|
||||
|
||||
@@ -5,13 +5,17 @@ import { ElfNotesService } from "../../application/binaryDiagnostics/ElfNotesSer
|
||||
import {
|
||||
AnalysisInputError,
|
||||
AnalysisResourceConstraintError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { projectAnalysisError } from "../../domain/analysisErrorProjection.js";
|
||||
import { elfNotesSchema } from "../../domain/native/elfNotes.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import {
|
||||
compressedElfNoteFixture,
|
||||
elfNote,
|
||||
elfNoteFixture,
|
||||
elfNoteTablePastEnd,
|
||||
elfExtendedSectionCountFixture,
|
||||
} from "./ElfNotes.fixture.js";
|
||||
import { ElfNotesProvider } from "./ElfNotesProvider.js";
|
||||
|
||||
@@ -48,6 +52,25 @@ describe("ELF note provider failures", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("reports valid extended section numbering as an unsupported target", async () => {
|
||||
const root = await createTestTempDirectory(
|
||||
"rea-elf-note-extended-numbering-",
|
||||
);
|
||||
const path = join(root, "extended-count.elf");
|
||||
await writeFile(path, elfExtendedSectionCountFixture());
|
||||
const result = await inspect(path);
|
||||
expect(result.ok).toBe(false);
|
||||
if (result.ok) throw new Error("Expected an unsupported target");
|
||||
expect(result.error).toBeInstanceOf(AnalysisUnsupportedTargetError);
|
||||
expect(result.error.message).toContain(
|
||||
"ELF extended numbering is unsupported",
|
||||
);
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
message: expect.stringContaining("ELF extended numbering is unsupported"),
|
||||
});
|
||||
});
|
||||
|
||||
it("names the note text budget instead of host memory", async () => {
|
||||
const root = await createTestTempDirectory("rea-elf-note-text-budget-");
|
||||
const path = join(root, "program");
|
||||
@@ -88,4 +111,41 @@ describe("ELF note provider failures", () => {
|
||||
expect(projected.remediation.action).toContain("max_file_bytes");
|
||||
expect(projected.remediation.action).not.toMatch(/available host memory/u);
|
||||
});
|
||||
|
||||
it("reports and admits compressed names through max_file_bytes", async () => {
|
||||
const bytes = compressedElfNoteFixture();
|
||||
const sectionOffset = Number(bytes.readBigUInt64LE(40));
|
||||
const nameHeader = sectionOffset + bytes.readUInt16LE(62) * 64;
|
||||
const namesOffset = Number(bytes.readBigUInt64LE(nameHeader + 24));
|
||||
const decodedNamesBytes = Number(bytes.readBigUInt64LE(namesOffset + 8));
|
||||
const minimumMaxFileBytes = bytes.length + decodedNamesBytes;
|
||||
const root = await createTestTempDirectory("rea-elf-note-name-budget-");
|
||||
const path = join(root, "compressed-names.elf");
|
||||
await writeFile(path, bytes);
|
||||
|
||||
const refused = await inspect(path, { max_file_bytes: bytes.length + 1 });
|
||||
expect(refused.ok).toBe(false);
|
||||
if (refused.ok) throw new Error("Expected section-name admission refusal");
|
||||
expect(refused.error).toBeInstanceOf(AnalysisResourceConstraintError);
|
||||
if (!(refused.error instanceof AnalysisResourceConstraintError))
|
||||
throw refused.error;
|
||||
expect(refused.error).toMatchObject({
|
||||
resource: "memory",
|
||||
reportedLimits: {
|
||||
max_file_bytes: bytes.length + 1,
|
||||
minimum_required_bytes: String(minimumMaxFileBytes),
|
||||
},
|
||||
retryable: false,
|
||||
});
|
||||
expect(projectAnalysisError(refused.error).remediation.action).toContain(
|
||||
"max_file_bytes",
|
||||
);
|
||||
|
||||
const admitted = await inspect(path, {
|
||||
max_file_bytes: minimumMaxFileBytes,
|
||||
});
|
||||
if (!admitted.ok) throw admitted.error;
|
||||
const report = elfNotesSchema.parse(admitted.value.normalized_result);
|
||||
expect(report.tables[0]?.name).toBe(".note.gnu.build-id");
|
||||
});
|
||||
});
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisInputError,
|
||||
AnalysisResourceConstraintError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { fixedAllowanceOptions } from "../../domain/fixedAllowanceRemediation.js";
|
||||
import {
|
||||
@@ -26,7 +27,11 @@ import {
|
||||
import { err, ok } from "../../domain/result.js";
|
||||
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
|
||||
import { ELF_NOTES_PROVIDER } from "../../application/InvestigationProviders.js";
|
||||
import { parseElfNotes } from "./ElfNoteParser.js";
|
||||
import { BinaryFormatFailure } from "../metadata/BinaryContainer.js";
|
||||
import {
|
||||
ElfNoteSectionNameBudgetFailure,
|
||||
parseElfNotes,
|
||||
} from "./ElfNoteParser.js";
|
||||
|
||||
const OPERATION = "inspect_elf_notes";
|
||||
|
||||
@@ -84,7 +89,7 @@ export class ElfNotesProvider implements ElfNotesPort {
|
||||
}
|
||||
|
||||
const NOTE_TEXT_BUDGET =
|
||||
"The ELF note text, hash, scan, and decompression budget bounded by max_file_bytes";
|
||||
"The ELF note text, hash, and scan budget fixed by the admitted image length plus decoded section names";
|
||||
|
||||
const inputBudgetRemediation = (
|
||||
boundary: "file-size" | "capacity",
|
||||
@@ -98,10 +103,47 @@ const inputBudgetRemediation = (
|
||||
? `The image exceeds the ${String(MAX_INSPECT_ELF_NOTE_ENTRIES)}-entry note-table ceiling this operation accepts; select a smaller artifact.`
|
||||
: `Raise max_entries above ${String(input.max_entries)} or select a smaller artifact; the limit is the caller-selected note entry budget.`;
|
||||
|
||||
const sectionNameBudgetRemediation = (
|
||||
cause: ElfNoteSectionNameBudgetFailure,
|
||||
): string =>
|
||||
cause.minimumMaxFileBytes <= BigInt(MAX_INSPECT_ELF_NOTE_FILE_BYTES)
|
||||
? `Raise max_file_bytes to at least ${String(cause.minimumMaxFileBytes)} bytes to admit the source and decoded ELF section names.`
|
||||
: `The source and decoded ELF section names require ${String(cause.minimumMaxFileBytes)} bytes, above this operation's ${String(MAX_INSPECT_ELF_NOTE_FILE_BYTES)}-byte maximum; select an image with a smaller section-name table.`;
|
||||
|
||||
const elfNotesFailurePrimary = (
|
||||
cause: unknown,
|
||||
input: InspectElfNotesInput,
|
||||
) => {
|
||||
if (cause instanceof ElfNoteSectionNameBudgetFailure)
|
||||
return new AnalysisResourceConstraintError(
|
||||
OPERATION,
|
||||
"memory",
|
||||
cause.message,
|
||||
{
|
||||
max_file_bytes: cause.selectedMaxFileBytes,
|
||||
minimum_required_bytes: cause.minimumMaxFileBytes.toString(),
|
||||
},
|
||||
{
|
||||
cause,
|
||||
retryable: false,
|
||||
remediationAction: sectionNameBudgetRemediation(cause),
|
||||
},
|
||||
);
|
||||
if (cause instanceof BinaryFormatFailure)
|
||||
return cause.kind === "unsupported"
|
||||
? new AnalysisUnsupportedTargetError(
|
||||
OPERATION,
|
||||
input.path,
|
||||
cause.message,
|
||||
{ cause },
|
||||
)
|
||||
: new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["path"],
|
||||
reason: "invalid_format",
|
||||
message: cause.message,
|
||||
},
|
||||
]);
|
||||
if (cause instanceof ArtifactReaderFailure && cause.reason === "integrity")
|
||||
return new AnalysisArtifactChangedError(
|
||||
OPERATION,
|
||||
@@ -119,7 +161,7 @@ const elfNotesFailurePrimary = (
|
||||
if (cause.reason === "limit") {
|
||||
// Caller-selected read and entry budgets name the bound to raise.
|
||||
// The note text, hash, and scan ceiling equals the admitted image
|
||||
// length, so host memory cannot raise it.
|
||||
// length plus decoded section names, so host memory cannot raise it.
|
||||
const noteTextBudget = cause.boundary === "memory";
|
||||
return new AnalysisResourceConstraintError(
|
||||
OPERATION,
|
||||
|
||||
@@ -20,7 +20,7 @@ import {
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { constants } from "node:fs";
|
||||
import { access, realpath, writeFile } from "node:fs/promises";
|
||||
import { extname, isAbsolute, join } from "node:path";
|
||||
import { basename, isAbsolute, join } from "node:path";
|
||||
import {
|
||||
createAnalysisExecution,
|
||||
type AnalysisExecution,
|
||||
@@ -177,10 +177,9 @@ export class SigrokRecordingProvider implements SignalRecordingPort {
|
||||
phase = "workspace";
|
||||
root = await this.createRuntime();
|
||||
this.#resources.retain(root, undefined);
|
||||
// Keep the selected file's suffix as private parser metadata. sigrok's
|
||||
// autodetection receives a filename, not just the copied bytes.
|
||||
const snapshotName =
|
||||
"path" in input ? `recording${extname(input.path)}` : "recording";
|
||||
// The private root is fresh; reuse the basename so its parser-relevant
|
||||
// suffix is preserved without growing a valid source name.
|
||||
const snapshotName = "path" in input ? basename(input.path) : "recording";
|
||||
const snapshotPath = join(root.path, snapshotName);
|
||||
const execute = async (
|
||||
args: readonly string[],
|
||||
|
||||
@@ -8,11 +8,14 @@ export type SigrokSelectorRejection = AnalysisInputIssue & {
|
||||
|
||||
/** Caller selectors that a sigrok command loads, keyed by their input field. */
|
||||
export type SigrokSelection =
|
||||
| Pick<DecodeSignalRecordingInput, "decoders" | "annotations">
|
||||
| Pick<
|
||||
DecodeSignalRecordingInput,
|
||||
"decoders" | "annotations" | "input_format"
|
||||
>
|
||||
| Pick<InspectSignalDecodersInput, "decoder">;
|
||||
|
||||
const rejectedSelector = (
|
||||
field: "decoder" | "decoders" | "annotations",
|
||||
field: "decoder" | "decoders" | "annotations" | "input_format",
|
||||
producerMessage: string,
|
||||
recovery: string,
|
||||
): SigrokSelectorRejection => ({
|
||||
@@ -27,6 +30,8 @@ const HELP_RECOVERY =
|
||||
"Omit decoder to list installed decoder IDs, optionally filtered by query, then request help for one of those IDs.";
|
||||
const CHANNEL_RECOVERY =
|
||||
"Set each decoder channel target to a channel present in the recording; inspect_signal_decoders shows the selected decoder's channel IDs.";
|
||||
const INPUT_FORMAT_RECOVERY =
|
||||
"Use a supported sigrok input format name, or omit input_format to let sigrok-cli autodetect the recording.";
|
||||
|
||||
/**
|
||||
* sigrok-cli loads selected decoders before decoding or showing help, so an
|
||||
@@ -94,6 +99,14 @@ export const parseSigrokSelectorRejection = (
|
||||
? undefined
|
||||
: rejectedSelector("decoder", refusal, HELP_RECOVERY);
|
||||
}
|
||||
const inputFormat = selection.input_format?.split(":")[0] ?? "";
|
||||
const inputFormatRefusal = `Error: unknown input module '${inputFormat}'.`;
|
||||
if (inputFormat !== "" && lines.has(`cli: ${inputFormatRefusal}`))
|
||||
return rejectedSelector(
|
||||
"input_format",
|
||||
inputFormatRefusal,
|
||||
INPUT_FORMAT_RECOVERY,
|
||||
);
|
||||
// sigrok-cli stacks decoders with "," and appends options with ":".
|
||||
for (const entry of selection.decoders.split(",")) {
|
||||
const refusal = unknownDecoderRefusal(lines, entry.split(":")[0] ?? "");
|
||||
|
||||
@@ -3,6 +3,7 @@ import { z } from "zod";
|
||||
import { jsonParts } from "../domain/jsonSerialization.js";
|
||||
import {
|
||||
SQLITE_ROW_LIMIT_DEFAULT,
|
||||
sqliteSchemaHasObjectDeclaration,
|
||||
sqliteSchemaScope,
|
||||
type InspectSqliteDatabaseInput,
|
||||
type SqliteDatabase,
|
||||
@@ -312,6 +313,25 @@ const readSchema = (
|
||||
selectedName === undefined || selected === undefined
|
||||
? undefined
|
||||
: shadowParentOf(selectedName, selected);
|
||||
// sqlite_schema is a queryable ordinary table in PRAGMA inventory, but has
|
||||
// no self-row in its stored declaration table. Keep these row-source facts
|
||||
// separate from schema.tables rather than manufacturing a declaration.
|
||||
const selectedRowSource =
|
||||
input.table === undefined ? undefined : metadata.get(input.table);
|
||||
if (
|
||||
input.table !== undefined &&
|
||||
!sqliteSchemaHasObjectDeclaration(input.table) &&
|
||||
selectedRowSource?.type === "table"
|
||||
)
|
||||
selectedTable = {
|
||||
name: selectedRowSource.name,
|
||||
kind: selectedRowSource.type,
|
||||
columns: readColumns(db, selectedRowSource.name, {
|
||||
budget: new OutputBudget(),
|
||||
encoding,
|
||||
selectedColumns: input.columns,
|
||||
}),
|
||||
};
|
||||
// NOCASE folds only ASCII letters, as SQLite does when it resolves the
|
||||
// table named by an index or trigger. Fold only for an ordinary table whose
|
||||
// rows can be selected; a selection refused later keeps the exact filter,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -61,3 +62,78 @@ cliTest(
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"projects nested APK v1 filename candidates as inventory metadata through CLI and MCP",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory(
|
||||
"rea-android-signing-candidates-",
|
||||
);
|
||||
const environment = { HOME: root, USERPROFILE: root };
|
||||
const { call } = await connectLocalToolsMcp(environment);
|
||||
const entries = [
|
||||
"META-INF/CERT.SF",
|
||||
"META-INF/CERT.EC",
|
||||
"META-INF/keys/NESTED.SF",
|
||||
"META-INF/keys/NESTED.EC",
|
||||
"META-INF/keys/readme.txt",
|
||||
"assets/not-signing.SF",
|
||||
];
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter());
|
||||
await writer.add("AndroidManifest.xml", new TextReader("fixture"));
|
||||
for (const entry of entries)
|
||||
await writer.add(entry, new TextReader(`inventory:${entry}`));
|
||||
const path = join(root, "signing-candidates.apk");
|
||||
await writeFile(path, await writer.close());
|
||||
|
||||
const inspected = await cli.run({
|
||||
arguments: ["inspect-artifact", path, "--json"],
|
||||
environment,
|
||||
});
|
||||
expect(inspected.exitCode, inspected.stderr).toBe(0);
|
||||
const input = { inventory_evidence: [parseEvidence(inspected.json)] };
|
||||
const projected = await cli.run({
|
||||
arguments: [
|
||||
"project-android-application-graph",
|
||||
JSON.stringify(input),
|
||||
"--json",
|
||||
],
|
||||
environment,
|
||||
});
|
||||
expect(projected.exitCode, projected.stderr).toBe(0);
|
||||
const cliEvidence = parseEvidence(projected.json);
|
||||
const cliGraph = androidApplicationProjectionResultSchema.parse(
|
||||
cliEvidence.normalized_result,
|
||||
);
|
||||
const mcp = await call("project_android_application_graph", input);
|
||||
expect(mcp.isError).not.toBe(true);
|
||||
const mcpEvidence = parseEvidence(mcp.structuredContent);
|
||||
const mcpGraph = androidApplicationProjectionResultSchema.parse(
|
||||
mcpEvidence.normalized_result,
|
||||
);
|
||||
|
||||
const expectedPaths = [
|
||||
"META-INF/CERT.EC",
|
||||
"META-INF/CERT.SF",
|
||||
"META-INF/keys/NESTED.EC",
|
||||
"META-INF/keys/NESTED.SF",
|
||||
];
|
||||
expect(cliGraph.components.signing.map(({ path }) => path).sort()).toEqual(
|
||||
expectedPaths,
|
||||
);
|
||||
expect(mcpGraph.components.signing).toEqual(cliGraph.components.signing);
|
||||
for (const component of cliGraph.components.signing)
|
||||
expect(component.sha256).toBe(
|
||||
createHash("sha256")
|
||||
.update(`inventory:${component.path}`)
|
||||
.digest("hex"),
|
||||
);
|
||||
expect(JSON.stringify(cliGraph.components.signing)).not.toContain(
|
||||
"readme.txt",
|
||||
);
|
||||
expect(JSON.stringify(cliGraph.components.signing)).not.toContain(
|
||||
"not-signing.SF",
|
||||
);
|
||||
expect(mcpEvidence).toEqual(cliEvidence);
|
||||
},
|
||||
);
|
||||
@@ -1,9 +1,10 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { toolContract } from "../../src/contracts/toolContracts.js";
|
||||
import {
|
||||
compressedElfNoteFixture,
|
||||
elfExtendedSectionCountFixture,
|
||||
elfNoteFixture,
|
||||
} from "../../src/native/elf/ElfNotes.fixture.js";
|
||||
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
|
||||
@@ -83,3 +84,22 @@ cliTest("reports a truncated ELF as invalid input", async ({ cli }) => {
|
||||
details: { operation: "inspect_elf_notes" },
|
||||
});
|
||||
});
|
||||
|
||||
cliTest(
|
||||
"reports valid extended numbering as unsupported without mutation",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-elf-notes-extended-cli-");
|
||||
const path = join(root, "extended-count.elf");
|
||||
await writeFile(path, elfExtendedSectionCountFixture());
|
||||
const before = await readFile(path);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-elf-notes", path, "--json"],
|
||||
});
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.json).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
message: expect.stringContaining("ELF extended numbering is unsupported"),
|
||||
});
|
||||
expect(await readFile(path)).toEqual(before);
|
||||
},
|
||||
);
|
||||
@@ -216,6 +216,50 @@ it("lists CArchive members and the Python version through inspect-artifact", asy
|
||||
});
|
||||
});
|
||||
|
||||
it("classifies a PyInstaller PE carrier when its signature straddles the probe prefix", async () => {
|
||||
const directory = await createTestTempDirectory("rea-pyinstaller-far-pe-");
|
||||
const path = join(directory, "hello.exe");
|
||||
await writeFile(
|
||||
path,
|
||||
cArchiveExecutable({
|
||||
carrier: "pe",
|
||||
peHeaderOffset: 8_190,
|
||||
layout: "2.1",
|
||||
pythonVersion: 314,
|
||||
pythonLibrary: "libpython3.14.dylib",
|
||||
members: [
|
||||
{
|
||||
name: "healthy.txt",
|
||||
typeCode: "x",
|
||||
payload: Buffer.from("healthy sibling"),
|
||||
compress: false,
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const result = await execute(process.execPath, [
|
||||
cli,
|
||||
"inspect-artifact",
|
||||
path,
|
||||
"--json",
|
||||
]);
|
||||
const inspection = artifactInspectionResultSchema.parse(
|
||||
parseEvidence(JSON.parse(result.stdout)).normalized_result,
|
||||
);
|
||||
expect(inspection.subject.root_format).toBe("pyinstaller");
|
||||
const inventory = inspection.substeps[0]?.evidence.normalized_result;
|
||||
expect(inventory?.manifest.root_format).toBe("pyinstaller");
|
||||
expect(inventory?.coverage.status).toBe("complete");
|
||||
expect(
|
||||
inventory?.occurrences.find(
|
||||
({ logical_path }) => logical_path === "healthy.txt",
|
||||
),
|
||||
).toMatchObject({
|
||||
hash_status: "verified",
|
||||
format_metadata: { source_name: "healthy.txt" },
|
||||
});
|
||||
});
|
||||
|
||||
it("extracts CArchive members through extract-artifact", async () => {
|
||||
const path = await onefile();
|
||||
const result = await execute(process.execPath, [
|
||||
|
||||
@@ -891,3 +891,65 @@ cliTest.runIf(SQLITE_NATIVE_LIMITS_AVAILABLE)(
|
||||
expect(await readFile(path)).toEqual(before);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest.runIf(SQLITE_NATIVE_LIMITS_AVAILABLE)(
|
||||
"reads sqlite_schema rows through CLI and MCP in every schema scope",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-sqlite-schema-table-");
|
||||
const path = join(root, "selected.db");
|
||||
const writer = new DatabaseSync(path);
|
||||
try {
|
||||
writer.exec("CREATE TABLE records(id INTEGER)");
|
||||
} finally {
|
||||
writer.close();
|
||||
}
|
||||
const original = await readFile(path);
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
for (const schema_scope of [
|
||||
"inventory",
|
||||
"selected_table",
|
||||
"database",
|
||||
] as const) {
|
||||
const result = await cli.run({
|
||||
arguments: [
|
||||
"inspect-sqlite-database",
|
||||
path,
|
||||
"--table",
|
||||
"sqlite_schema",
|
||||
"--column",
|
||||
"name",
|
||||
"--column",
|
||||
"type",
|
||||
"--schema-scope",
|
||||
schema_scope,
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(result.exitCode, result.stdout + result.stderr).toBe(0);
|
||||
expect(result.json).toMatchObject({
|
||||
normalized_result: {
|
||||
schema: { scope: schema_scope },
|
||||
rows: {
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
values: [
|
||||
[
|
||||
{ type: "text", value: "records" },
|
||||
{ type: "text", value: "table" },
|
||||
],
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
const response = await call("inspect_sqlite_database", {
|
||||
path,
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
schema_scope,
|
||||
});
|
||||
expect(response.isError, JSON.stringify(response)).not.toBe(true);
|
||||
expect(response.structuredContent).toEqual(result.json);
|
||||
}
|
||||
expect(await readFile(path)).toEqual(original);
|
||||
},
|
||||
);
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -175,6 +176,59 @@ describe("Android application projection", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("projects APK v1 metadata candidates at any META-INF depth", async () => {
|
||||
const root = await createTestTempDirectory("rea-android-v1-metadata-");
|
||||
const path = join(root, "NestedSigning.apk");
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter());
|
||||
const names = [
|
||||
"META-INF/CERT.SF",
|
||||
"META-INF/CERT.EC",
|
||||
"META-INF/keys/CERT.SF",
|
||||
"META-INF/keys/CERT.EC",
|
||||
"META-INF/keys/readme.txt",
|
||||
"META-INF/keys/MANIFEST.MF",
|
||||
"assets/not-signing.SF",
|
||||
"meta-inf/lower.SF",
|
||||
];
|
||||
for (const entry of names)
|
||||
await writer.add(entry, new TextReader(`content:${entry}`));
|
||||
await writeFile(path, await writer.close());
|
||||
|
||||
const inventory = parseEvidence(
|
||||
await runProviderAnalysis(path, "inventory_artifact", {}),
|
||||
);
|
||||
const projection = androidApplicationProjectionResultSchema.parse(
|
||||
requireSuccessfulProjection(
|
||||
projectAndroidApplicationEvidence({ inventory_evidence: [inventory] }),
|
||||
).normalized_result,
|
||||
);
|
||||
|
||||
expect(projection.components.signing.map(({ path }) => path).sort()).toEqual([
|
||||
"META-INF/CERT.EC",
|
||||
"META-INF/CERT.SF",
|
||||
"META-INF/keys/CERT.EC",
|
||||
"META-INF/keys/CERT.SF",
|
||||
]);
|
||||
for (const name of [
|
||||
"META-INF/CERT.SF",
|
||||
"META-INF/CERT.EC",
|
||||
"META-INF/keys/CERT.SF",
|
||||
"META-INF/keys/CERT.EC",
|
||||
]) {
|
||||
const component = projection.components.signing.find(
|
||||
({ path }) => path === name,
|
||||
);
|
||||
expect(component).toMatchObject({
|
||||
path: name,
|
||||
artifact_id: expect.stringMatching(/^art_/u),
|
||||
sha256: createHash("sha256").update(`content:${name}`).digest("hex"),
|
||||
});
|
||||
}
|
||||
expect(projection.limitations).toContain(
|
||||
"Manifest, resource, signing, and bytecode semantics require a dedicated Android provider; this projection reports exact inventory paths and hashes only.",
|
||||
);
|
||||
});
|
||||
|
||||
describe("APK family without a root manifest", () => {
|
||||
const missingManifest =
|
||||
"No root AndroidManifest.xml occurrence was observed. The APK family comes from the .apk suffix of a ZIP archive, so these bytes may not be an APK.";
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js";
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { inventoryArtifact } from "../../fixtures/artifactInventory.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { buildGodotPck } from "../../fixtures/godotPck.js";
|
||||
|
||||
const fatMachO = (sliceOffset: number): Buffer => {
|
||||
const thin = Buffer.alloc(48);
|
||||
thin.writeUInt32LE(0xfeedfacf, 0);
|
||||
thin.writeUInt32LE(0x0100000c, 4);
|
||||
thin.writeUInt32LE(0, 8);
|
||||
thin.writeUInt32LE(2, 12);
|
||||
|
||||
const bytes = Buffer.alloc(sliceOffset + thin.length);
|
||||
bytes.writeUInt32BE(0xcafebabe, 0);
|
||||
bytes.writeUInt32BE(1, 4);
|
||||
bytes.writeUInt32BE(0x0100000c, 8);
|
||||
bytes.writeUInt32BE(0, 12);
|
||||
bytes.writeUInt32BE(sliceOffset, 16);
|
||||
bytes.writeUInt32BE(thin.length, 20);
|
||||
bytes.writeUInt32BE(Math.log2(sliceOffset), 24);
|
||||
thin.copy(bytes, sliceOffset);
|
||||
return bytes;
|
||||
};
|
||||
|
||||
const zipWithMember = async (bytes: Buffer): Promise<Buffer> => {
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter());
|
||||
await writer.add("native.fat", new Uint8ArrayReader(bytes), { level: 0 });
|
||||
return Buffer.from(await writer.close());
|
||||
};
|
||||
|
||||
for (const sliceOffset of [4_096, 16_384]) {
|
||||
for (const container of ["zip", "pck"] as const) {
|
||||
it(`retains a FAT slice header at offset ${sliceOffset} in ${container} member nodes`, async () => {
|
||||
const root = await createTestTempDirectory("rea-fat-member-node-");
|
||||
const bytes = fatMachO(sliceOffset);
|
||||
const directPath = join(root, "direct.fat");
|
||||
await writeFile(directPath, bytes);
|
||||
const direct = await inventoryArtifact(directPath);
|
||||
const directNode = direct.nodes.find(
|
||||
({ artifact_id }) => artifact_id === direct.manifest.root_artifact_id,
|
||||
);
|
||||
expect(directNode?.format).toBe("mach-o-universal");
|
||||
|
||||
const containerPath = join(root, `fixture.${container}`);
|
||||
await writeFile(
|
||||
containerPath,
|
||||
container === "zip"
|
||||
? await zipWithMember(bytes)
|
||||
: buildGodotPck(3, [{ path: "native.fat", data: bytes }]),
|
||||
);
|
||||
const inventory = await inventoryArtifact(containerPath);
|
||||
const occurrence = inventory.occurrences.find(
|
||||
({ logical_path }) => logical_path === "native.fat",
|
||||
);
|
||||
expect(occurrence).toMatchObject({
|
||||
artifact_format: "mach-o-universal",
|
||||
hash_status: "verified",
|
||||
artifact_id: direct.manifest.root_artifact_id,
|
||||
});
|
||||
const node = inventory.nodes.find(
|
||||
({ artifact_id }) => artifact_id === occurrence?.artifact_id,
|
||||
);
|
||||
expect(node).toMatchObject({
|
||||
artifact_id: direct.manifest.root_artifact_id,
|
||||
sha256: direct.manifest.root_sha256,
|
||||
format: "mach-o-universal",
|
||||
});
|
||||
expect(inventory.coverage).toEqual({
|
||||
scope: "container-entries",
|
||||
status: "complete",
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,15 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { Uint8ArrayReader, Uint8ArrayWriter, ZipWriter } from "@zip.js/zip.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { classifyArtifactBytesAtPath } from "../../fixtures/artifactClassification.js";
|
||||
import { ARTIFACT_CLASSIFICATION_PREFIX_BYTES } from "../../../src/artifacts/ArtifactClassificationProbe.js";
|
||||
import { scanCanonicalArtifactInventory } from "../../fixtures/artifactInventory.js";
|
||||
import {
|
||||
inventoryArtifact,
|
||||
scanCanonicalArtifactInventory,
|
||||
} from "../../fixtures/artifactInventory.js";
|
||||
import { artifactInventoryResultSchema } from "../../../src/domain/artifactGraph.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
|
||||
@@ -82,6 +86,30 @@ describe("MZ artifact classification", () => {
|
||||
expect.arrayContaining(["dos-mz", "pe"]),
|
||||
);
|
||||
});
|
||||
|
||||
it("retains a PE signature straddling the bounded prefix in ZIP member evidence", async () => {
|
||||
const root = await createTestTempDirectory("rea-mz-far-pe-");
|
||||
const bytes = peFixture(ARTIFACT_CLASSIFICATION_PREFIX_BYTES - 2);
|
||||
const writer = new ZipWriter(new Uint8ArrayWriter());
|
||||
await writer.add("native.exe", new Uint8ArrayReader(bytes), { level: 0 });
|
||||
const path = join(root, "fixture.zip");
|
||||
await writeFile(path, Buffer.from(await writer.close()));
|
||||
|
||||
const inventory = await inventoryArtifact(path);
|
||||
const occurrence = inventory.occurrences.find(
|
||||
({ logical_path }) => logical_path === "native.exe",
|
||||
);
|
||||
expect(occurrence).toMatchObject({
|
||||
artifact_format: "pe",
|
||||
hash_status: "verified",
|
||||
});
|
||||
expect(
|
||||
inventory.nodes.find(
|
||||
({ artifact_id }) => artifact_id === occurrence?.artifact_id,
|
||||
),
|
||||
).toMatchObject({ format: "pe", size: bytes.length });
|
||||
expect(inventory.coverage.status).toBe("complete");
|
||||
});
|
||||
});
|
||||
|
||||
const dosFixture = (): Buffer => {
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
import { writeFile } from "node:fs/promises";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { toolContract } from "../../../src/contracts/toolContracts.js";
|
||||
import { createServer } from "../../../src/server/createServer.js";
|
||||
import { createTestBinarySession } from "../../fixtures/binarySession.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import { elfNoteFixture } from "../../../src/native/elf/ElfNotes.fixture.js";
|
||||
import {
|
||||
elfExtendedSectionCountFixture,
|
||||
elfNoteFixture,
|
||||
} from "../../../src/native/elf/ElfNotes.fixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { mcpTest, type TestMcp } from "../../support/mcp/mcpFixture.js";
|
||||
|
||||
@@ -68,3 +71,30 @@ mcpTest("requires an absolute path for inspect_elf_notes", async ({ mcp }) => {
|
||||
await session.close();
|
||||
}
|
||||
});
|
||||
|
||||
mcpTest(
|
||||
"reports valid extended numbering as unsupported without mutation",
|
||||
async ({ mcp }) => {
|
||||
const { client, session } = await connect(mcp);
|
||||
try {
|
||||
const root = await createTestTempDirectory("rea-elf-notes-extended-mcp-");
|
||||
const path = join(root, "extended-count.elf");
|
||||
await writeFile(path, elfExtendedSectionCountFixture());
|
||||
const before = await readFile(path);
|
||||
const response = await client.callTool({
|
||||
name: "inspect_elf_notes",
|
||||
arguments: { path },
|
||||
});
|
||||
expect(response.isError).toBe(true);
|
||||
expect(parseMcpToolError(response).error).toMatchObject({
|
||||
code: "unsupported_target",
|
||||
message: expect.stringContaining(
|
||||
"ELF extended numbering is unsupported",
|
||||
),
|
||||
});
|
||||
expect(await readFile(path)).toEqual(before);
|
||||
} finally {
|
||||
await session.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
@@ -7,7 +7,7 @@ import { projectAnalysisError } from "../../../src/domain/analysisErrorProjectio
|
||||
import { OwnedCommandFailure } from "../../../src/process/OwnedCommand.js";
|
||||
import { it, expect, onTestFinished } from "vitest";
|
||||
import { writeFile, readdir, chmod, symlink } from "node:fs/promises";
|
||||
import { extname, join } from "node:path";
|
||||
import { basename, join } from "node:path";
|
||||
import { readStableArtifact } from "../../../src/artifacts/readStableArtifact.js";
|
||||
import { SigrokRecordingProvider } from "../../../src/signals/SigrokRecordingProvider.js";
|
||||
import { SignalRecordingService } from "../../../src/application/signals/SignalRecordingService.js";
|
||||
@@ -26,22 +26,11 @@ import { Cli } from "incur";
|
||||
import { registerSignalCommands } from "../../../src/cli/signalCommands.js";
|
||||
import { silentLogger } from "../../../src/logger.js";
|
||||
|
||||
// A command-boundary simulator: real process ownership/filesystem, not real sigrok coverage.
|
||||
const fixture = async (mode = "ok", failCleanup = false) => {
|
||||
const directory = await createTestTempDirectory("rea-signal-test-");
|
||||
const path = join(
|
||||
directory,
|
||||
mode === "csv-autodetect" ? "capture.csv" : "recorded bytes.vcd",
|
||||
);
|
||||
const snapshotName = `recording${extname(path)}`;
|
||||
await writeFile(
|
||||
path,
|
||||
mode === "csv-autodetect" ? "CLK,MOSI\n0,0\n1,0\n" : "recording bytes",
|
||||
);
|
||||
const script = join(directory, "producer.mjs");
|
||||
await writeFile(
|
||||
script,
|
||||
`import {readFileSync, writeFileSync, chmodSync, truncateSync} from 'node:fs';
|
||||
const producerSource = (mode: string, snapshotName: string): string =>
|
||||
`import {readFileSync, writeFileSync, chmodSync, truncateSync} from 'node:fs';
|
||||
const inputFileArgument = process.argv.find((argument) => argument.startsWith('--input-file='));
|
||||
const snapshotName = inputFileArgument?.slice('--input-file='.length);
|
||||
if (inputFileArgument !== undefined && inputFileArgument !== ${JSON.stringify(`--input-file=${snapshotName}`)}) throw Error('wrong snapshot name');
|
||||
if (process.argv.includes('--version')) {
|
||||
if (${JSON.stringify(mode)} === 'invalid-version-stdout') process.stdout.write(Buffer.concat([Buffer.from('sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3\\n'), Buffer.from([0xff])]));
|
||||
else console.log(${JSON.stringify(mode)} === 'bad-version' ? 'sigrok-cli 0.7.1' : 'sigrok-cli 0.7.2\\nlibsigrokdecode 0.5.3');
|
||||
@@ -50,6 +39,7 @@ if (process.argv.includes('--version')) {
|
||||
}
|
||||
else if (${JSON.stringify(mode)} === 'bad-version') console.log('unexpected version fallback');
|
||||
else if (${JSON.stringify(mode)} === 'nonzero') { console.error('invalid decoder'); process.exitCode = 1; }
|
||||
else if (${JSON.stringify(mode)} === 'unknown-input-format') { if (!process.argv.includes('--input-format=missingformat')) console.error('fixture: selected input format was not forwarded'); console.error("cli: Error: unknown input module 'missingformat'."); process.exitCode = 1; }
|
||||
else if (${JSON.stringify(mode)} === 'unknown-decoder') { console.error("srd: ModuleNotFoundError: Failed to load decoder nosuchdecoder: import by name failed: No module named 'nosuchdecoder'\\nFailed to load protocol decoder nosuchdecoder."); process.exitCode = 1; }
|
||||
else if (${JSON.stringify(mode)} === 'broken-decoder') { console.error("srd: ModuleNotFoundError: Failed to load decoder uart: import by name failed: No module named 'serial'\\nFailed to load protocol decoder uart."); process.exitCode = 1; }
|
||||
else if (${JSON.stringify(mode)} === 'unknown-annotation') { console.error("Annotation 'nosuchclass' not found for protocol decoder 'uart'."); process.exitCode = 1; }
|
||||
@@ -62,18 +52,38 @@ else if (${JSON.stringify(mode)} === 'invalid-stderr') { process.stdout.write('1
|
||||
else if (${JSON.stringify(mode)} === 'literal-replacement') process.stdout.write('10-20 uart-1: �\\n');
|
||||
else if (${JSON.stringify(mode)} === 'invalid-catalog-stdout' && process.argv.includes('--list-supported')) process.stdout.write(Buffer.concat([Buffer.from('Supported protocol decoders:\\n uart '), Buffer.from([0xff])]));
|
||||
else if (${JSON.stringify(mode)} === 'invalid-help-stdout' && process.argv.includes('--show')) process.stdout.write(Buffer.concat([Buffer.from('ID: uart\\nOptions:\\n'), Buffer.from([0xfe])]));
|
||||
else if (${JSON.stringify(mode)} === 'malformed-copy') { chmodSync(${JSON.stringify(snapshotName)}, 0o600); writeFileSync(${JSON.stringify(snapshotName)}, 'changed'); }
|
||||
else if (${JSON.stringify(mode)} === 'oversized-copy') { chmodSync(${JSON.stringify(snapshotName)}, 0o600); truncateSync(${JSON.stringify(snapshotName)}, 32 * 1024 * 1024 + 1); }
|
||||
else if (${JSON.stringify(mode)} === 'malformed-copy') { chmodSync(snapshotName, 0o600); writeFileSync(snapshotName, 'changed'); }
|
||||
else if (${JSON.stringify(mode)} === 'oversized-copy') { chmodSync(snapshotName, 0o600); truncateSync(snapshotName, 32 * 1024 * 1024 + 1); }
|
||||
else if (${JSON.stringify(mode)} === 'line-budget') process.stdout.write('ID: uart' + '\\n'.repeat(65536));
|
||||
else if (${JSON.stringify(mode)} === 'overflow') process.stdout.write('x'.repeat(5 * 1024 * 1024));
|
||||
else if (${JSON.stringify(mode)} === 'wait') { console.log('ready'); setInterval(() => {}, 1000); }
|
||||
else if (${JSON.stringify(mode)} === 'many') process.stdout.write(Array.from({length:90},(_,i)=>'line '+(i+1)+'\\r\\n').join(''));
|
||||
else if (process.argv.includes('--list-supported')) { console.log('Supported hardware drivers:\\n ignored Device\\nSupported protocol decoders:\\n uart Universal Asynchronous Receiver/Transmitter\\n i2c Inter-Integrated Circuit\\n'); console.error('catalog warning'); }
|
||||
else if (process.argv.includes('--show')) console.log('ID: uart\\nOptions:\\n- baudrate: Baud rate (default 115200)\\n- parity: Parity');
|
||||
else if (${JSON.stringify(mode)} === 'csv-autodetect') { if (!process.argv.includes('--input-file=recording.csv') || readFileSync(${JSON.stringify(snapshotName)}, 'utf8') !== 'CLK,MOSI\\n0,0\\n1,0\\n') throw Error('CSV suffix or copy bytes lost'); console.log('10-20 spi-1: 1'); }
|
||||
else { if(readFileSync(${JSON.stringify(snapshotName)}, 'utf8') !== 'recording bytes') throw Error('wrong copy'); console.log('10-20 uart-1: 42'); console.error('producer warning'); }
|
||||
`,
|
||||
else if (${JSON.stringify(mode)} === 'csv-autodetect') { if (!process.argv.includes(${JSON.stringify(`--input-file=${snapshotName}`)}) || readFileSync(snapshotName, 'utf8') !== 'CLK,MOSI\\n0,0\\n1,0\\n') throw Error('CSV suffix or copy bytes lost'); console.log('10-20 spi-1: 1'); }
|
||||
else if (${JSON.stringify(mode)} === 'long-basename') { if (!process.argv.includes('--input-format=vcd') || readFileSync(snapshotName, 'utf8') !== 'recording bytes') throw Error('long source basename or explicit format was lost'); console.log('10-20 uart-1: 42'); }
|
||||
else { if(readFileSync(snapshotName, 'utf8') !== 'recording bytes') throw Error('wrong copy'); console.log('10-20 uart-1: 42'); console.error('producer warning'); }
|
||||
`;
|
||||
|
||||
// A command-boundary simulator: real process ownership/filesystem, not real sigrok coverage.
|
||||
const fixture = async (
|
||||
mode = "ok",
|
||||
failCleanup = false,
|
||||
recordingName?: string,
|
||||
) => {
|
||||
const directory = await createTestTempDirectory("rea-signal-test-");
|
||||
const path = join(
|
||||
directory,
|
||||
recordingName ??
|
||||
(mode === "csv-autodetect" ? "capture.csv" : "recorded bytes.vcd"),
|
||||
);
|
||||
const snapshotName = basename(path);
|
||||
await writeFile(
|
||||
path,
|
||||
mode === "csv-autodetect" ? "CLK,MOSI\n0,0\n1,0\n" : "recording bytes",
|
||||
);
|
||||
const script = join(directory, "producer.mjs");
|
||||
await writeFile(script, producerSource(mode, snapshotName));
|
||||
await chmod(script, 0o700);
|
||||
const configuredCommand =
|
||||
mode === "symlink" ? join(directory, "configured-tool") : script;
|
||||
@@ -205,7 +215,7 @@ it("pages literal output without changing decoder arguments or normalizing text"
|
||||
expect(report.output_window.returned_lines).toBe(count);
|
||||
expect(report.tool.arguments).toEqual([
|
||||
"--dont-scan",
|
||||
"--input-file=recording.vcd",
|
||||
`--input-file=${basename(f.path)}`,
|
||||
"--protocol-decoders=uart",
|
||||
"--protocol-decoder-samplenum",
|
||||
]);
|
||||
@@ -248,7 +258,7 @@ it("decodes an immutable copy, preserves options and warnings, and releases its
|
||||
expect(report.selection.input_format).toBeNull();
|
||||
expect(report.tool.arguments).toEqual([
|
||||
"--dont-scan",
|
||||
"--input-file=recording.vcd",
|
||||
"--input-file=recorded bytes.vcd",
|
||||
"--protocol-decoders=uart:rx=RX:baudrate=9600",
|
||||
"--protocol-decoder-samplenum",
|
||||
"--protocol-decoder-annotations=uart=rx-data",
|
||||
@@ -259,24 +269,49 @@ it("decodes an immutable copy, preserves options and warnings, and releases its
|
||||
]);
|
||||
});
|
||||
|
||||
it("preserves a CSV filename hint through the private autodetection copy", async () => {
|
||||
const f = await fixture("csv-autodetect");
|
||||
it.each(["capture.csv", ".capture.csv", ".csv"])(
|
||||
"preserves the CSV filename hint from %s through the private copy",
|
||||
async (recordingName) => {
|
||||
const f = await fixture("csv-autodetect", false, recordingName);
|
||||
const result = await f.service.decode({
|
||||
path: f.path,
|
||||
decoders: "spi:clk=CLK:mosi=MOSI",
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
const report = signalRecordingSchema.parse(result.value.normalized_result);
|
||||
expect(report.selection).toMatchObject({
|
||||
path: f.path,
|
||||
input_format: null,
|
||||
});
|
||||
expect(report.tool.arguments).toEqual([
|
||||
"--dont-scan",
|
||||
`--input-file=${recordingName}`,
|
||||
"--protocol-decoders=spi:clk=CLK:mosi=MOSI",
|
||||
"--protocol-decoder-samplenum",
|
||||
]);
|
||||
},
|
||||
);
|
||||
|
||||
it("stages a long source basename without expanding it before decoding", async () => {
|
||||
const recordingName = `a.${"x".repeat(246)}`;
|
||||
expect(Buffer.byteLength(recordingName)).toBe(248);
|
||||
const f = await fixture("long-basename", false, recordingName);
|
||||
const result = await f.service.decode({
|
||||
path: f.path,
|
||||
decoders: "spi:clk=CLK:mosi=MOSI",
|
||||
decoders: "uart",
|
||||
input_format: "vcd",
|
||||
});
|
||||
if (!result.ok) throw result.error;
|
||||
|
||||
const report = signalRecordingSchema.parse(result.value.normalized_result);
|
||||
expect(report.selection).toMatchObject({
|
||||
path: f.path,
|
||||
input_format: null,
|
||||
});
|
||||
expect(report.tool.arguments).toEqual([
|
||||
"--dont-scan",
|
||||
"--input-file=recording.csv",
|
||||
"--protocol-decoders=spi:clk=CLK:mosi=MOSI",
|
||||
"--protocol-decoder-samplenum",
|
||||
]);
|
||||
expect(report.artifact.path).toBe(f.path);
|
||||
expect(report.selection.input_format).toBe("vcd");
|
||||
expect(report.annotations_text).toBe("10-20 uart-1: 42\n");
|
||||
expect(report.tool.arguments).toContain(`--input-file=${recordingName}`);
|
||||
expect(f.calls()).toBe(2);
|
||||
expect(
|
||||
(await readdir(f.directory)).filter((name) => name.startsWith("decoder-")),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
@@ -311,23 +346,32 @@ it.each([
|
||||
{ decoders: "uart:rx=RX,nosuchdecoder" },
|
||||
"decoders",
|
||||
"Failed to load protocol decoder nosuchdecoder.",
|
||||
"inspect_signal_decoders",
|
||||
],
|
||||
[
|
||||
"unknown-annotation",
|
||||
{ decoders: "uart", annotations: "uart=rx-data:nosuchclass" },
|
||||
"annotations",
|
||||
"Annotation 'nosuchclass' not found for protocol decoder 'uart'.",
|
||||
"inspect_signal_decoders",
|
||||
],
|
||||
[
|
||||
"unknown-input-format",
|
||||
{ decoders: "uart", input_format: "missingformat" },
|
||||
"input_format",
|
||||
"Error: unknown input module 'missingformat'.",
|
||||
"omit input_format",
|
||||
],
|
||||
] as const)(
|
||||
"reports sigrok's %s refusal as invalid caller input",
|
||||
async (mode, selection, field, refusal) => {
|
||||
async (mode, selection, field, refusal, recovery) => {
|
||||
const f = await fixture(mode);
|
||||
const result = await f.service.decode({ path: f.path, ...selection });
|
||||
if (result.ok) throw new Error("Expected selector refusal");
|
||||
expect(result.error._tag).toBe("AnalysisInputError");
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "invalid_request",
|
||||
message: expect.stringContaining("inspect_signal_decoders"),
|
||||
message: expect.stringContaining(recovery),
|
||||
details: {
|
||||
issues: [
|
||||
{
|
||||
@@ -351,6 +395,95 @@ it.each([
|
||||
},
|
||||
);
|
||||
|
||||
it("reports an unknown selected input format through CLI and MCP", async () => {
|
||||
const f = await fixture("unknown-input-format");
|
||||
const refusal = "Error: unknown input module 'missingformat'.";
|
||||
const issueMessage =
|
||||
`sigrok-cli rejected the input_format selector: ${refusal} ` +
|
||||
"Use a supported sigrok input format name, or omit input_format to let sigrok-cli autodetect the recording.";
|
||||
const harness = await createApplicationMcpHarness({
|
||||
serverOptions: { environment: {}, signalRecording: f.service },
|
||||
});
|
||||
onTestFinished(harness.close);
|
||||
const { error } = parseMcpToolError(
|
||||
await harness.client.callTool({
|
||||
name: "decode_signal_recording",
|
||||
arguments: {
|
||||
path: f.path,
|
||||
decoders: "uart",
|
||||
input_format: "missingformat",
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(error).toMatchObject({
|
||||
code: "invalid_request",
|
||||
category: "invalid_input",
|
||||
message: expect.stringContaining(issueMessage),
|
||||
details: {
|
||||
issues: [
|
||||
{
|
||||
path: ["input_format"],
|
||||
reason: "invalid_value",
|
||||
message: expect.stringContaining(issueMessage),
|
||||
},
|
||||
],
|
||||
captured_output: {
|
||||
exit_code: 1,
|
||||
stderr: expect.stringContaining(refusal),
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(JSON.stringify(error)).not.toContain(
|
||||
"fixture: selected input format was not forwarded",
|
||||
);
|
||||
|
||||
const cli = Cli.create("signal-input-format-refusal-test", { sync: false });
|
||||
registerSignalCommands(cli, silentLogger, {}, f.service);
|
||||
let stdout = "";
|
||||
await cli.serve(
|
||||
[
|
||||
"decode-signal-recording",
|
||||
f.path,
|
||||
"--decoders",
|
||||
"uart",
|
||||
"--input_format",
|
||||
"missingformat",
|
||||
"--format",
|
||||
"json",
|
||||
],
|
||||
{
|
||||
stdout: (text) => {
|
||||
stdout += text;
|
||||
},
|
||||
exit: () => {},
|
||||
},
|
||||
);
|
||||
expect(JSON.parse(stdout)).toMatchObject({
|
||||
code: "invalid_request",
|
||||
message: expect.stringContaining(issueMessage),
|
||||
details: {
|
||||
issues: [
|
||||
{
|
||||
path: ["input_format"],
|
||||
reason: "invalid_value",
|
||||
message: expect.stringContaining(issueMessage),
|
||||
},
|
||||
],
|
||||
captured_output: {
|
||||
exit_code: 1,
|
||||
stderr: expect.stringContaining(refusal),
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(stdout).not.toContain(
|
||||
"fixture: selected input format was not forwarded",
|
||||
);
|
||||
expect(f.calls()).toBe(4);
|
||||
expect(
|
||||
(await readdir(f.directory)).filter((name) => name.startsWith("decoder-")),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("projects sigrok's zero-exit missing-channel refusal through CLI and MCP with partial output", async () => {
|
||||
const f = await fixture("unknown-channel");
|
||||
const refusal = 'No channel with name "NONEXISTENT" found.';
|
||||
@@ -620,6 +753,8 @@ it.each([
|
||||
["unknown-decoder", { decoder: "uart" }],
|
||||
["unknown-decoder", {}],
|
||||
["unknown-annotation", { decoders: "uart", annotations: "uart=rx-data" }],
|
||||
["unknown-input-format", { decoders: "uart", input_format: null }],
|
||||
["unknown-input-format", { decoders: "uart", input_format: "vcd" }],
|
||||
] as const)(
|
||||
"keeps %s outside caller selectors %j as a provider failure",
|
||||
async (mode, selection) => {
|
||||
@@ -813,7 +948,7 @@ it.each(["oversized-copy", "oversized-tool"])(
|
||||
path:
|
||||
mode === "oversized-tool"
|
||||
? join(f.directory, "producer.mjs")
|
||||
: expect.stringMatching(/decoder-.*[/\\]recording\.vcd$/u),
|
||||
: expect.stringMatching(/decoder-.*[/\\]recorded bytes\.vcd$/u),
|
||||
});
|
||||
expect(
|
||||
(await readdir(f.directory)).filter((name) =>
|
||||
|
||||
@@ -282,6 +282,37 @@ it("keeps unrelated catalog names outside the selected-table reply budget", asyn
|
||||
).toThrowError(expect.objectContaining({ reason: "output-limit" }));
|
||||
});
|
||||
|
||||
it.each(["inventory", "selected_table", "database"] as const)(
|
||||
"reads sqlite_schema rows using PRAGMA row-source facts in %s scope",
|
||||
async (schema_scope) => {
|
||||
const path = await createDatabase(
|
||||
"CREATE TABLE records(id INTEGER); INSERT INTO records VALUES (1);",
|
||||
);
|
||||
const result = inspectSqliteDatabaseSnapshot(path, {
|
||||
path,
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
schema_scope,
|
||||
});
|
||||
|
||||
expect(result.rows).toMatchObject({
|
||||
table: "sqlite_schema",
|
||||
columns: ["name", "type"],
|
||||
values: [
|
||||
[
|
||||
{ type: "text", value: "records" },
|
||||
{ type: "text", value: "table" },
|
||||
],
|
||||
],
|
||||
});
|
||||
expect(result.schema.tables).not.toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ name: "sqlite_schema" }),
|
||||
]),
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects malformed schema bytes instead of returning replacement text", async () => {
|
||||
const path = await createDatabase(
|
||||
"CREATE TABLE records(value TEXT /*schema-marker*/)",
|
||||
|
||||
+2
-1
@@ -9,9 +9,10 @@ export const classifyArtifactBytesAtPath = (
|
||||
) => {
|
||||
const probe = createAsarHeaderSignatureProbe();
|
||||
probe.write(bytes);
|
||||
return classifyArtifactContent(
|
||||
const { kind, format } = classifyArtifactContent(
|
||||
path,
|
||||
{ prefix: bytes, asarHeader: probe.finish(fileSize) },
|
||||
fileSize,
|
||||
);
|
||||
return { kind, format };
|
||||
};
|
||||
Vendored
+8
-3
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
|
||||
|
||||
export interface GodotPckMember {
|
||||
readonly path: string;
|
||||
readonly data: string;
|
||||
readonly data: string | Uint8Array;
|
||||
readonly flags?: number;
|
||||
readonly md5?: Buffer;
|
||||
readonly size?: bigint;
|
||||
@@ -35,10 +35,12 @@ export const buildGodotPck = (
|
||||
): Buffer => {
|
||||
const data: Buffer[] = [];
|
||||
const offsets: number[] = [];
|
||||
const memberBytes: Buffer[] = [];
|
||||
let dataBytes = 0;
|
||||
for (const member of members) {
|
||||
offsets.push(dataBytes);
|
||||
const bytes = pad4(Buffer.from(member.data));
|
||||
memberBytes.push(Buffer.from(member.data));
|
||||
data.push(bytes);
|
||||
dataBytes += bytes.length;
|
||||
}
|
||||
@@ -50,8 +52,11 @@ export const buildGodotPck = (
|
||||
pckU32(path.length),
|
||||
path,
|
||||
pckU64(member.offset ?? BigInt(offsets[index] ?? 0)),
|
||||
pckU64(member.size ?? BigInt(Buffer.byteLength(member.data))),
|
||||
member.md5 ?? createHash("md5").update(member.data).digest(),
|
||||
pckU64(member.size ?? BigInt(memberBytes[index]?.length ?? 0)),
|
||||
member.md5 ??
|
||||
createHash("md5")
|
||||
.update(memberBytes[index] ?? "")
|
||||
.digest(),
|
||||
pckU32(member.flags ?? 0),
|
||||
];
|
||||
}),
|
||||
|
||||
Reference in new issue
Block a user