Compare commits

...
Author SHA1 Message Date
Kaoru0822andClaude Opus 5.5 8a8efb8226 fix(javascript): keep unselected sources out of cancellation failures
With source_paths, the analysis loop checked cancellation before it
recognized an outside-analysis-scope file. Cancellation arriving after the
last selected source therefore recorded the next unselected file as a
cancelled analysis failure, and the completion loop skipped it as already
analyzed.

Give out-of-selection classification one owner, used by the loop, the
interruption handler and the completion loop. The cancellation is still
reported as the interruption.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 21:36:10 +00:00
Dery Ferdikaandmorluto 1d4c4eb030 fix(adb): reject unsafe host file names when pulling one device file (#1962)
pull_adb_file derived its host file name from the last path segment of a
caller-supplied device path and joined it to the output directory without
checking that the name is usable on the host. A device path ending in a
separator, a dot segment, a control character, a Windows reserved device
name, a trailing dot or space, or a ':' alternate-data-stream marker could
resolve to an unexpected host path or fail opaquely after the transfer.

The single-file path now derives the name with POSIX semantics and refuses
any non-portable segment as invalid_request before creating a directory or
running adb pull. This matches the package path's pending host-name policy
and never renames a device-reported file.

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 05:33:40 +08:00
Nithilan Kumaranandmorluto 9eee709f0c fix(wasm): accept unpadded WABT section rows (#1963)
* fix(wasm): accept unpadded WABT section rows

WABT 1.0.42 right-aligns section names in a nine-column field. DataCount
fills that field, so its objdump row has no leading whitespace. Accept
that legal row. Header recognition, the required field shape, and section
range checks still reject malformed output.

Fixes #1956

* fix(ci): refresh MIPS expectations and Windows test paths

Assert admitted soft-float ABI metadata and its committed storage policy through the public target resolver, retain FP64 rejection, and point the curated Windows lane at the moved filesystem reconstruction tests.

Validation: reproduced all four prior failures; 97 focused tests and check:ci pass. Native Windows and real Ghidra execution were not performed locally.

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 05:29:04 +08:00
morluto beb80a07ba test(cli): consolidate schema coverage and assert input refusals
Remove 30 redundant truthy-JSON schema checks. Validate advertised groups and property parity across every primary command, retain required-field checks, and assert structured malformed-input errors. Exercise JEB positional validation rather than an unknown flag.

Validation: 13 public CLI adapter tests, combined checks, and temporary-directory hygiene. Host configuration refusal coverage from main is preserved.
2026-10-12 05:24:33 +08:00
morluto 108b5f9425 fix(native): reject ambiguous otool load-command framing
Validate Mach command counts, byte totals, scalar uniqueness, section counts, and build-tool counts before claiming exhaustive evidence. Preserve raw output in typed parsing failures and distinguish dylib commands from the dynamic loader. Correct inconsistent handcrafted fixtures and label their provenance.

Validation: 38 native tests, combined checks, and temporary-directory hygiene. Multiline dylib-name ambiguity was reproduced with the host Apple otool; regression workflows use captured output and injected command runners.
2026-10-12 05:24:29 +08:00
morluto a8d4984404 fix(browser): preserve uncertainty in malformed network observations
Require observed request method and initiator metadata instead of inventing defaults. Preserve valid redirect history while excluding later events for malformed reused IDs. Keep unavailable or invalid encoded lengths unknown and retain valid zero values.

Validation: 57 CDP boundary tests plus temporary-directory hygiene and combined checks. Protocol events come from the fake CDP server; no live browser verification is claimed.
2026-10-12 05:24:24 +08:00
morluto c002b89e2d fix(snapshot): bind replayed subjects to the selected target path
Rebind display paths only for matching artifact identities in primitive and composed replay. Preserve semantic Evidence IDs, payloads, and historical snapshot provenance. Classify snapshot filesystem workflows as boundary tests.

Validation: cache, direct workflow, MCP replay, and Evidence tests; combined checks and temporary-directory hygiene. Provider interactions are simulated for replay verification.
2026-10-12 05:24:20 +08:00
morluto 2260887f20 fix(artifacts): close mounted directory resources before DMG detach
Wait for directory operations, close owned streams and traversal handles, and retain the mounted reader when cleanup fails so close can retry before detach. Consolidate stable-artifact filesystem regressions into the boundary suite.

Validation: 35 artifact tests and temporary-directory hygiene, including real hdiutil diagnostics. Directory-close and traversal races use injected faults.
2026-10-12 05:24:15 +08:00
morluto c2407689cb fix(setup): preserve skill files and report rollback failures
Track attempted writes, preserve original content and modes from one regular-file observation, and continue rollback after individual failures. Return tagged failure details and retained backup paths through setup.

Validation: setup filesystem and recovery tests, combined compile/typecheck/lint/format/metadata/architecture checks. Filesystem fault injection covers backup, ambiguous publication, and rollback failures.
2026-10-12 05:24:10 +08:00
39418ea173 fix(adb): share host configuration refusals with apktool (#1954)
The ADB provider, which the apktool provider mirrors, had the same two
defects: a relative REA_ADB_PATH escaped as UNKNOWN because selection
ran before the try that maps configuration failures, and a missing adb
binary projected generic target advice instead of its reason.

Move the configuration-failure-to-refusal mapping into one
providerConfigurationRefusal helper used by both providers, resolve the
ADB selection inside the mapped try, and share the CLI refusal assertion.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 05:09:13 +08:00
7d92d93a55 fix(managed): share first-match member list ownership (#1955)
Inventory attribute lookups and the members inspector each interpreted
TypeDef member-list ownership. The members lookup binary-searched ordered
ranges but fell back to a per-member scan of every type when malformed
lists were unordered, which was quadratic (8 s for 100,000 types).

Both now use firstMatchListOwners, one near-linear first-match index over
one-based list positions. Remove the ordered/fallback split and the
production declaringType scan, which remains only as the test oracle. The
inventory reserves only its retained index.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 05:06:41 +08:00
morluto eb97da3388 fix(artifacts): retain ASAR resources through cleanup failures
Give snapshot capture, header admission, packed members, and container streams one file-handle owner. Serialize adoption with reader shutdown, retain failed closes for retry, and remove snapshots only after descriptor cleanup succeeds.

Cover unread members, pending opens, zero-byte members, and paired format/cleanup failures. Keep intentionally unrecoverable test fixtures hygienic and strengthen managed identity and process cleanup-authority expectations.

Validation: full npm run check; 99 targeted tests and temporary-directory hygiene; exact discovery of 877 test files. No real vendor-provider verification was performed.
2026-10-12 05:05:21 +08:00
Aurelio Bernalandmorluto 290d301ec7 feat(ghidra): admit relocatable MIPS32r2 ELF objects (#1950)
* feat(ghidra): admit soft-float MIPS32r2 ELF with integer-register float storage

Soft-float o32 executables (Linux kernel builds, bare-metal firmware) were
refused because their ABI flags record declares no CPR1 and FP ABI 3.
Ghidra's MIPS32 compiler specifications model only hard-float o32, so typed
float/double signatures (DWARF) would be bound to $f12/$f14/$f0 and the
decompiler loses the real dataflow: `return sel ? a : b` decompiled to
`return in_f0;`, and callers passed phantom FPU inputs.

Admit ELF32 ET_EXEC o32 MIPS32r2 targets whose ABI record declares soft-float
with no CPR1, and commit `mips_soft_float_storage: o32-integer-registers-v1`
in their analysis profile only (hard-float profile digests are unchanged).
ReaGhidraMipsSoftFloat.java runs after auto-analysis and, for each function
with a float/double formal type, either keeps the types with the storage
Ghidra's o32 model assigns to a same-size integer (pair alignment, stack
spill), or, for auto-parameters such as hidden struct-return pointers, swaps
float types for same-size integers under dynamic storage. No function is left
on hard-float storage. The report is written last; the provider client
requires it exactly when the profile commits the policy and refuses the
session before any operation otherwise.

Verification: real Ghidra 12.1.4 lane (verify:ghidra:mips) now builds
hard- and soft-float fixtures in both byte orders and checks exact
parameter/return storage through inspect_native_api, caller constants,
snapshot profile and Evidence outcome; test:fast, verify:package and an
installed-tarball soft-float run pass.

* fix(ghidra): model soft-float MIPS as a prototype for every typed callable

Review feedback on #1949: rewriting storage per Function left external
functions and indirect calls through typed function pointers on Ghidra's
hard-float o32 prototype, which still read $f12/$f14/$f0 while the session
reported success. A negative control confirms it: without the change,
`(*rea_mips_soft_callback)(8.5f, 9.5f, sel)` decompiles to
`(*rea_mips_soft_callback)(in_f12, in_f14, sel)`.

ReaGhidraMipsSoftFloat.java now derives a `__rea_soft_float` prototype from
the program's own default o32 model (encoded, float pentries and
float-consuming rules removed, so byte-order-specific aggregate rules and the
hidden struct-return pointer are kept), installs it as a specification
extension, and assigns it with dynamic storage to every non-thunk function,
external function and function-pointer definition whose formal types use
float or double. Thunks follow their targets. Float types are kept
everywhere, so the integer-retype fallback is removed. The script fails
without writing its report if the derived prototype or any reassigned
function still uses an FPU register.

The policy becomes `o32-soft-float-prototype-v1` with a
{functions, definitions} report. The soft-float fixture adds an indirect
call through a typed pointer; the real-Ghidra lane requires its constants
with no FPU inputs and the struct return's `float scale` in $a1.

* test(ghidra): pin ODDSPREG acceptance for soft-float MIPS ABI records

Review on #1949 suggested requiring flags1 == 0 for soft-float, since MIPS_AFL_FLAGS1_ODDSPREG names CPR1 registers. GCC 5.2 kernel modules, LLVM/Rust firmware and Clang -msoft-float objects all record flags1 = 1 with FP ABI soft-float: it is the MIPS32 o32 toolchain default and selects nothing without CPR1. Keep accepting it, say why at the check, and test that soft-float with ODDSPREG is admitted while any other general flag is refused.

* feat(ghidra): admit relocatable MIPS32r2 ELF objects

Linux MIPS kernel modules (.ko) and other ELF32 o32 MIPS32r2 ET_REL objects
were refused by the Ghidra MIPS lane. Ghidra's ELF loader already places each
allocated section and applies REL relocations; this admits ET_REL alongside
ET_EXEC (ET_DYN and PRX stay refused) under its own support lane,
`elf32-rel-o32-arch32r2-standard-v1`, so executable targets keep theirs.
Relocatable sessions carry limitations: addresses are analysis coordinates,
undefined symbols live in Ghidra's EXTERNAL block, and HI16 relocations report
PARTIAL because Ghidra applies them with their paired LO16.

Verification: new `verify:ghidra:mips-relocatable` real-Ghidra lane builds
-c -G0 objects in both byte orders plus a soft-float -mlong-calls module
variant. GNU readelf supplies sections, symbols and REL entries; every
relocation in an allocated section must encode S + A in the loaded word
(S from Ghidra placement, A from original bytes), with HI16/LO16 checked as
the effective address each LO16 forms, and coverage of a shared HI16, a LO16
carry, external code/data and R_MIPS_32 is asserted. The executable lane,
unit suites pass, and on 30 real .ko files 54,972 relocations matched (93 reference empty sections and are not checkable).

* fix(ghidra): verify every MIPS HI16 and archive-typed soft-float externals

Review feedback on #1950:

- The relocation oracle kept one pending HI16, so a second HI16 before the
  LO16 that resolves both overwrote the first, and unrelated relocations
  cleared it; the earlier site's high half and carry were never compared
  with S + A. Every HI16 now waits for the next LO16 of its symbol, which
  checks all of them; later LO16s reuse those HI16s, and a HI16 left
  unresolved at a section end fails. On 30 real soft-float kernel modules,
  24 contain several HI16s per LO16; all 54,972 placed relocations match.
- External callees with float prototypes: the relocatable fixture now calls
  powf, whose prototype Ghidra's C library archive applies by name. Every
  variant must decompile `powf(1.5,2.5)` without FPU inputs, and the
  soft-float module variant must report that __rea_soft_float covered it
  (with the external step removed it decompiles as `powf(param_1,param_2)`).
  `-fno-builtin` keeps the call from being folded.

Also updates the module variant's expected soft-float policy to
`o32-soft-float-prototype-v1`.

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 05:02:55 +08:00
Aurelio Bernal 5e219de441 feat(ghidra): admit soft-float MIPS32r2 ELF with integer-register float storage (#1949)
* feat(ghidra): admit soft-float MIPS32r2 ELF with integer-register float storage

Soft-float o32 executables (Linux kernel builds, bare-metal firmware) were
refused because their ABI flags record declares no CPR1 and FP ABI 3.
Ghidra's MIPS32 compiler specifications model only hard-float o32, so typed
float/double signatures (DWARF) would be bound to $f12/$f14/$f0 and the
decompiler loses the real dataflow: `return sel ? a : b` decompiled to
`return in_f0;`, and callers passed phantom FPU inputs.

Admit ELF32 ET_EXEC o32 MIPS32r2 targets whose ABI record declares soft-float
with no CPR1, and commit `mips_soft_float_storage: o32-integer-registers-v1`
in their analysis profile only (hard-float profile digests are unchanged).
ReaGhidraMipsSoftFloat.java runs after auto-analysis and, for each function
with a float/double formal type, either keeps the types with the storage
Ghidra's o32 model assigns to a same-size integer (pair alignment, stack
spill), or, for auto-parameters such as hidden struct-return pointers, swaps
float types for same-size integers under dynamic storage. No function is left
on hard-float storage. The report is written last; the provider client
requires it exactly when the profile commits the policy and refuses the
session before any operation otherwise.

Verification: real Ghidra 12.1.4 lane (verify:ghidra:mips) now builds
hard- and soft-float fixtures in both byte orders and checks exact
parameter/return storage through inspect_native_api, caller constants,
snapshot profile and Evidence outcome; test:fast, verify:package and an
installed-tarball soft-float run pass.

* fix(ghidra): model soft-float MIPS as a prototype for every typed callable

Review feedback on #1949: rewriting storage per Function left external
functions and indirect calls through typed function pointers on Ghidra's
hard-float o32 prototype, which still read $f12/$f14/$f0 while the session
reported success. A negative control confirms it: without the change,
`(*rea_mips_soft_callback)(8.5f, 9.5f, sel)` decompiles to
`(*rea_mips_soft_callback)(in_f12, in_f14, sel)`.

ReaGhidraMipsSoftFloat.java now derives a `__rea_soft_float` prototype from
the program's own default o32 model (encoded, float pentries and
float-consuming rules removed, so byte-order-specific aggregate rules and the
hidden struct-return pointer are kept), installs it as a specification
extension, and assigns it with dynamic storage to every non-thunk function,
external function and function-pointer definition whose formal types use
float or double. Thunks follow their targets. Float types are kept
everywhere, so the integer-retype fallback is removed. The script fails
without writing its report if the derived prototype or any reassigned
function still uses an FPU register.

The policy becomes `o32-soft-float-prototype-v1` with a
{functions, definitions} report. The soft-float fixture adds an indirect
call through a typed pointer; the real-Ghidra lane requires its constants
with no FPU inputs and the struct return's `float scale` in $a1.

* test(ghidra): pin ODDSPREG acceptance for soft-float MIPS ABI records

Review on #1949 suggested requiring flags1 == 0 for soft-float, since MIPS_AFL_FLAGS1_ODDSPREG names CPR1 registers. GCC 5.2 kernel modules, LLVM/Rust firmware and Clang -msoft-float objects all record flags1 = 1 with FP ABI soft-float: it is the MIPS32 o32 toolchain default and selects nothing without CPR1. Keep accepting it, say why at the check, and test that soft-float with ODDSPREG is admitted while any other general flag is refused.
2026-10-12 04:54:38 +08:00
morluto 5ba1642a7d test: strengthen archive and resource cleanup assertions
Replace duplicate path-helper checks with real ZIP inventory coverage and replace global memory sampling with bounded-chunk assertions while retaining 64 MiB digest coverage. Await artifact client cleanup, preserve Hopper lease ownership during teardown, and expose typed application failures.
2026-10-12 04:53:27 +08:00
morluto 804c287e44 test: classify filesystem application workflows as boundaries 2026-10-12 04:53:10 +08:00
morluto 918713af37 fix(artifacts): bind directory reads to captured file state
Capture exact filesystem identity and nanosecond timestamps, validate descriptor/path state before and after streaming, and enforce the captured byte count. Bound growing files to their captured size plus one byte using precise positional reads; keep unrepresentable display sizes explicitly unknown.

Reuse the captured-state comparison around Apple assetutil and retain existing reader cleanup ownership. Validation: focused reader and affected consumer suites, full static/architecture checks, and a compiled directory-read probe on Bun 1.3.14 passed. Includes same-size mutation, growth, truncation, cancellation, FIFO replacement, cleanup retry, ZIP identity forgery, ASAR and Mach-O coverage.
2026-10-12 04:45:17 +08:00
morluto a82e37c806 fix(browser): retain module resolver cleanup ownership
Serialize resolution with shutdown, retain the owned browser after close failures, and retry cleanup before another launch. Wire cleanup through the shared service and CLI/MCP lifecycles, preserving primary failures and completed native reports alongside cleanup diagnostics.

Separate the browser resolution workflow from lifecycle ownership. Validation: seven resolver source tests, eleven application tests, MCP shutdown retry, public CLI parity, and full static/architecture checks passed. Real Chromium behavior remains unverified.
2026-10-12 04:44:25 +08:00
morluto c2b56e3e1c test: classify filesystem and process workflows by boundary
Move artifact inventory, application projection, runtime reconciliation, and TODO scanner suites into their real boundary lanes. Extract Hopper process/socket cleanup cases while keeping injected session cleanup cases in composition.

Validation: all 56 focused tests passed; discovery finds all 876 test files exactly once. Test assertions and provider verification scope are unchanged.
2026-10-12 04:44:13 +08:00
morluto 12fa8fd9f9 fix(ida): recover cleanup after worker sessions are reaped
Accept the supervisor absent-session response only for the owned session, then independently verify session and private-input absence before releasing the workspace. Keep failed inventory verification retryable without resending a completed close.

Validation: 59 focused source tests passed, including live private-input peers, unavailable inventory, and mismatched session IDs. Real IDA engine behavior remains unverified. Fixes #1710.
2026-10-12 04:40:41 +08:00
morluto dc303a8ee8 test(javascript): classify artifact workflows as boundary tests
Move the six JavaScript application suites that read production artifacts or launch compiled analysis workers out of composition. Focused test planning now prepares their runtime prerequisites instead of treating them as source-only tests.

Validation: all six relocated suites and adjacent artifact/reference suites pass (70 tests across eight files); discovery finds every test exactly once.
2026-10-12 04:33:51 +08:00
morluto 3d83135987 fix(reference): verify symlink identity around target reads
Bracket symlink target observation with the identity captured during enumeration. Report replacements and disappearing links as changed, and keep cancellation and I/O mapping at one owner boundary.

Remove duplicate target-error wrappers and unreachable external-target handling. Validation: reference reader filesystem suite passes, including deterministic replacement during observation.
2026-10-12 04:33:47 +08:00
morluto 09b6828516 fix(process): cancel run-token reads during sampling
Propagate the sample AbortSignal through readProcessRunId to host environment inspection so cancellation can settle a blocked ownership-token read.

Validation: ownership and capture lifecycle suites pass, including an abort-waiting host regression. Native provider runtime behavior was not exercised.
2026-10-12 04:33:42 +08:00
morluto f64790bfc3 refactor(javascript): reuse validated source-map decoding
Pass typed map objects into the codec and reuse validated TraceMap instances for regular and indexed lookups. Remove whole-map serialization, duplicate leaf decoding, an array-copy wrapper, and the unused offset-validation mode in artifact declaration collection.

Validation: source-map and browser consumer suites (126 tests), declaration tests, and artifact reconstruction boundaries pass. Section overlap checks and raw source declarations remain intact.
2026-10-12 04:33:38 +08:00
morluto c42232f566 fix(adb): share actionable provider configuration refusals
Catch binary selection failures at the adapter boundary and share the domain error projection with apktool. Missing and relative adb paths now report capability_unavailable with host configuration guidance in both message and remediation.

Validation: ADB/apktool provider suites and public CLI refusal suites pass. No connected-device behavior changed or verified. Refs #1952.
2026-10-12 04:33:34 +08:00
morluto 52e76c5d39 fix(managed): share first-match member ownership indexes
Resolve fields, methods, and attribute constructors through one budgeted index per layout. Preserve malformed-list first-match ownership without per-member TypeDef scans; remove the superseded lookup implementations.

Validate pointer identity and malformed ranges through production parsers, including a 40,000-type/method case and allocation admission. Managed source tests and CLI/MCP pointer acceptance pass. Refs #1953.
2026-10-12 04:33:25 +08:00
1449e8e051 fix(mcp): provide native evidence paging continuation
Supply schema-validated inspect_analysis_view arguments after completed native function Evidence is recorded and exceeds MCP delivery capacity. Preserve the full canonical result and its provenance. Verify exact source paging, export and following-call usability through real Ghidra stdio MCP.

Fixes #1849

Co-authored-by: Minghang Chamling <133689405+cold-atom@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 04:26:05 +08:00
Kaoru0822andClaude Opus 5.5 7cdb7981c3 fix(apktool): name host requirements in configuration refusals (#1946)
A missing apktool launcher projected the generic "unavailable for the
current target" message and remediation, leaving the actual reason only
in details.reason. A relative REA_APKTOOL_COMMAND escaped as UNKNOWN
because command selection ran before the try that maps configuration
failures.

Resolve the selection inside that try for both execution and
availability, and pass the refusal reason as userMessage, as JADX and
the adapters fixed in #1224 do.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:13:23 +08:00
Mykola Dementievandmorluto 0ae38e1e31 fix(ida): preserve non-x86 call edges and workflow uncertainty (#1906)
* test(ida): expect callee edges from the call graph on non-x86 targets

* fix(ida): read callee edges from the call graph on non-x86 targets

* test(ida): pin address order of call graph additions

* docs(ida): describe the call graph source for non-x86 callees

* test(ida): pin the x86 baseline, the call graph reader and its failure modes

* fix(ida): check the call graph root, scope truncation to the root and degrade without callgraph

* test(ida): expect managed PE off the x86 path, possible-truncation wording and a complete ARM64 traversal

* fix(ida): keep managed PE and x86 reads off the call graph path and qualify truncation

* test(ida): answer only expected leaf roots and assert the requested call graph roots

* test(ida): expect a call graph the producer cannot give consistently to fail the callee read

* fix(ida): fail the callee read when the call graph cannot be obtained consistently

* fix(ida): preserve call edge authority and workflow uncertainty

* fix(ida): preserve explicit root call graph caps

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 04:10:22 +08:00
Kaoru0822andClaude Opus 5.5 c2cf7dcb86 fix(managed): index attribute constructor owners once per layout (#1943)
Each CustomAttribute whose constructor is a local MethodDef scanned the
TypeDef table to find the declaring type, so inspect_managed_artifact did
O(attributes x types) row decoding. A 584 KB assembly with 16,000 types
and 16,000 such attributes blocked for about 38 s.

Build a first-match owner index once per metadata layout. Already-owned
list positions are skipped, so overlapping malformed method lists keep
their previous first-match owner in near-linear time. The index is
charged to the decode budget, and a build failure is cached like the
member pointer lookup.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 04:09:50 +08:00
morlutoandhhang 14935e4d47 fix(javascript): preserve package and source-map boundary identity (#1939)
* fix(javascript): preserve bare package specifier suffixes

* fix(javascript): resolve ESM package self references

* fix(javascript): recover retained prefixed source maps

* test(javascript): cover suffixed ESM self references

* fix(javascript): retain legacy ESM package suffix behavior

* fix(javascript): reject legacy ESM directory and extension fallback

* refactor(javascript): consolidate package resolution and workflow coverage

* fix(javascript): report exact ESM package subpath misses

* fix(javascript): compare parsed names for ESM self references

* refactor(javascript): parse bare package requests once

* fix(javascript): derive ESM dependency lookup from package URLs

---------

Co-authored-by: hhang <169027046+xu91102@users.noreply.github.com>
2026-10-12 04:09:17 +08:00
85c1d2587a feat(javascript): analyze selected application sources
Analyze exact selected sources in first-request order before unrelated
static scopes, preserving full artifact identity and byte verification.
Retain selection in CLI/MCP Evidence and summaries, with explicit unknowns
and partial coverage for unselected sources. Keep default full analysis.

Fixes #1848

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 04:07:14 +08:00
morluto 0d8b728d24 docs(agents): require related brittleness cleanup in bug fixes 2026-10-12 03:58:24 +08:00
9878bb7296 feat(ghidra): apply validated analyzer overrides before analysis
Apply caller-selected analyzer opt-outs before Ghidra analysis, reject unknown or ineffective names, and commit raw/header imports to the same analyzer profile. Use supported x86 raw compiler specs and cover the real analyzer/readback and raw workflows.

Adopts the original implementation from #1800 with validated corrections. Refs #1783, #1836.

Co-authored-by: akushonkamen <74654766+akushonkamen@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 03:49:31 +08:00
12ee785e5e feat(website): follow the system dark mode and fix small-screen text (#1894)
* fix(website): keep text inside small screens and tidy small type and spacing

From a layout audit of the built pages:

- Long terminal prompts and code-header buttons wrap instead of being cut
  off at 320px (WCAG 1.4.10).
- Headings and table cells break long words, so the 1.4.12 text-spacing
  test no longer widens pages past 320px.
- Header and footer links get a 24px minimum height (2.5.8).
- Labels that rendered under 12px now render at 12px or more, and inline
  code never drops under 0.75rem.
- The home page's hero and definition rows share one column gap and align
  on the first baseline, and stacked boxes share one corner size.
- Highlight chips get room around the text, and the active run in a step
  listing is one band instead of a box per line.

Colors are unchanged.

🤖 Generated with Claude Code (Sonnet 5.5) 🦊

* feat(website): follow the system dark mode

The site was light only, so a reader with a dark system still got a white
page. Add one prefers-color-scheme: dark block to the stylesheet that swaps
the six color variables to Catppuccin Mocha (MIT) and overrides the fixed
colors that would break on a dark page: selection, buttons, tinted
highlights, flow nodes and arrows, and a visible border on the agent
terminal.

Add a dark theme-color tag beside the light one in the page template and
the 17 pages, and a Dark column to the style guide's color table.

Light colors are unchanged. The SVG figures follow in the next commit; the
blog page's inline strokes, the dinosaur game and the social card are left
as they are.

🤖 Generated with Claude Code (Sonnet 5.5) 🦊

* feat(website): give the five SVG figures a dark scheme

Pages load the figures with <img>, so page CSS cannot reach inside them.
Each figure now carries its own @media (prefers-color-scheme: dark) block
of attribute-selector overrides that map every light color to a Catppuccin
Mocha color, with one shared map across all five. No markup, labels,
shapes or positions change, and nothing uses !important.

Four figures also paint a mantle canvas so "Open figure" works on its own;
analyst-thinking.svg stays transparent, since it sits on the page rather
than in a figure box.

🤖 Generated with Claude Code (Sonnet 5.5) 🦊

* chore(website): refresh translation source references

The theme-color tags move the page lines that the catalogs cite, so
python3 scripts/website.py translations extract rewrites the source
references in messages.pot and the 18 PO files. No message text changes.
The same run moves the "Main navigation" entry to its new position in each
catalog.

🤖 Generated with Claude Code (Sonnet 5.5) 🦊

* fix(website): keep inline step highlights inline

The full-width band for the active step now applies only to code-step
blocks. The inline parameter highlights in the dx-ball and th04
signatures stay inline again.

🤖 Generated with Claude Code (Opus 5.5) 🦊

* test(website): check every page offers light and dark theme colors

Each authored reading page, and its built copy in every language,
must carry one theme-color tag for light and one for dark. The
stylesheet must keep its dark color scheme block. On the commit before
dark mode, this test fails on every page.

🤖 Generated with Claude Code (Opus 5.5) 🦊

* fix(website): draw the dinosaur game in the page colors

In dark mode the game box turned dark, but the game still drew with
its light colors, so the dinosaur, cacti and text fell to about 2:1
contrast. The canvas now reads five colors from CSS variables on
.dino-stage. Light values are the old ones, so light mode draws the
same pixels. Dark values are Catppuccin Mocha: 8.3:1 for the
dinosaur, 7.9:1 for cacti and text, 7.6:1 for the crash color.

The moved lines in dino-demo.js shift the translation catalog
references, so the catalogs are refreshed too (reference comments
only).

🤖 Generated with Claude Code (Opus 5.5) 🦊

* fix(website): recolor the Touhou flywheel arrows in dark mode

The inline flywheel SVG on the Touhou article hard-codes its stroke
colors, so on the dark page its connectors fell to about 2:1. Two
dark-only rules map those strokes to the page accent and muted colors
(7.8:1 and 7.4:1). Light mode is unchanged.

🤖 Generated with Claude Code (Opus 5.5) 🦊

* fix(website): redraw the dinosaur game when the system theme changes

The canvas reads its colors only when it draws, so an idle, paused or
crashed game kept the old palette after a theme switch. Each demo now
redraws on a prefers-color-scheme change.

The added lines shift the translation catalog references, so the
catalogs are refreshed too (reference comments only).

🤖 Generated with Claude Code (Opus 5.5) 🦊

---------

Co-authored-by: Claude Sonnet 5.5 <sonnet@cassie-pc>
Co-authored-by: Claude Opus 5.5 <opus@cassie-pc>
2026-10-12 03:17:49 +08:00
e59db2f5de fix(session): verify retained client liveness on open fast path
Verify retained provider operation health before reusing the same target and profile. Retire failed Hopper clients through existing owned cleanup, preserve healthy reuse and cancellation, and start replacement children lazily on the next query.

Refs #1719. Actual headless Linux Hopper child-death/reopen recovery and 85 related tests passed against the current-main merge tree; Ghidra/IDA liveness and the historical snapshot report remain outside this verification.

Co-authored-by: sx4im <sx4im@users.noreply.github.com>
Co-authored-by: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 03:16:13 +08:00
Lluís Ariño d802304c54 fix(install): describe automatic setup and guard truncated downloads (#1902)
* fix(install): describe automatic setup and guard truncated downloads

The help text said the installer only installs the CLI and that
`rea setup` must be run separately, but the installer starts
`rea setup` interactively whenever a terminal is available. Describe
that behaviour and the --no-setup/--no-prompt opt-out.

Wrap the installer body in `main` and call it on the last line, so a
truncated `curl | bash` download fails to parse instead of executing a
partial script. Behaviour is otherwise unchanged.

Add an installer scenario that runs a copy truncated right after the
npm install line and requires that it prints nothing and never invokes
npm; it fails against the previous script.

Validation: bash -n, shellcheck 0.10.0, installScript boundary tests.

* fix(install): require the closing token before running main

Wrapping the installer in main() kept a truncated download from running
a partial body, but the last line was a bare `main "$@"`. If the piped
download stopped right after `main` (or `main `), bash saw a complete
command at EOF and ran main with no positional parameters. Under the
documented `curl ... | bash -s -- --dry-run` (or --version / --no-setup)
path that silently dropped the caller's flags, so a dry run could resolve
the latest release, run npm, and start setup.

The call now sits in a brace group:

  {
    main "$@"
    exit $?
  }

Bash does not run the group until it has read the closing brace, so
every truncation inside it is a syntax error and nothing executes; a
truncation before the group leaves only function definitions. A comment
in install.sh records why the bare call is not enough.

Validation: a new boundary test pipes install.sh to `bash -s --
--dry-run` truncated at every byte offset from the end of main() to EOF
and asserts that npm and the rea binary are never invoked; offsets
inside the group must exit non-zero with a bash parse error and no
stdout, offsets before it must print nothing, and the complete group
must finish the dry run. Reproduced the original bug by cutting the old
script after `main`: the dry run invoked npm install. `bash -n` and
shellcheck 0.11.0 are clean.

* fix(install): wrap the whole script in the truncation guard

The previous guard only wrapped the final `main "$@"` invocation, so the
preamble streamed on the `curl ... | bash` path before any incomplete
compound command was open. A download cut off right after the bytes
`set` reached EOF with a complete, argument-less `set`, which prints every
shell variable, so inherited tokens could reach stdout.

The brace group now opens on line 2, directly after the shebang, and
closes on the last line, enclosing `set -euo pipefail`, the constants, the
functions and the final `main "$@"` / `exit $?`. Bash must therefore read
the whole file before running anything; any earlier cut is a syntax error
at EOF. `set -euo pipefail` still applies inside the group (checked
manually for `false`, an unbound variable, and a failing function body),
and shellcheck 0.11.0 and `bash -n` are clean.

The truncation test now pipes every prefix of install.sh, from byte 0 to
EOF (about 6,100 cuts, run 16 at a time with a minimal environment that
includes a sentinel variable), into `bash -s -- --dry-run`. Before the
group opens it must exit quietly; inside the group it must fail with a
syntax error; in every case stdout stays empty and neither npm nor rea is
invoked. Only the complete script performs the dry run.
2026-10-12 02:47:31 +08:00
YsopwnableandKaoru0822 0f4992fb9e perf(ghidra): compile bundle activator with bridge scripts (#1931)
Include the no-op OSGi lifecycle class in the packaged source bundle so
Ghidra does not generate and compile an extra activator afterward.
Check the selected activator in the real packaged Windows verifier.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 02:46:32 +08:00
Kaoru0822andClaude Opus 5.5 72489a9445 fix(javascript): report available parent result budget in coverage limits (#1915)
* fix(javascript): size the future static reserve by remaining sources

The parent reserved 80% of its result budget for future static facts
whenever any source remained, so one tiny later file could discard an
earlier file's completed semantic graph. Reserve at most one bounded
static checkpoint's retained size per remaining source, within the same
shared pool, so many remaining sources keep the previous protection.
Parent-budget rejections now report the effective admission budget and
explain that the CLI/MCP process heap, not max_heap_mb, bounds them.

Refs #1911

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): make parent budget remediation and its test runtime-aware

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): suggest a concrete parent heap size in remediation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(javascript): let parent budget rejections state their retained facts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:43:31 +08:00
morluto b6651cb99f fix(javascript): retain completed facts through cancellation (#1934) 2026-10-12 02:39:52 +08:00
morluto 0b2aa69531 Merge remote-tracking branch 'origin/main' into codex/pr-1899-remove-duplicate-schema-test 2026-10-12 02:38:57 +08:00
morluto b5b73019f9 test(web): remove duplicate path schema cases 2026-10-12 02:38:16 +08:00
Lluís Ariño d0edeec100 test(native): skip the inaccessible-directory diagnostic as root (#1900)
The artifact directory diagnostic makes a bundle member unreadable with
mode 000 and expects inventory_artifact to report its path and EACCES.
Root ignores the mode, so the read succeeds and the test fails when the
suite runs as uid 0 (for example in a container). Skip it there with the
same process.getuid?.() === 0 condition the DMG EACCES diagnostic uses.
2026-10-12 02:35:31 +08:00
morluto 3b490a743f test(apple): consolidate keyed archive integer verification (#1933) 2026-10-12 02:34:24 +08:00
morluto 3298219d8c docs(readme): remove CI badges from translations 2026-10-12 02:33:29 +08:00
morluto b07e54a10e docs(readme): remove CI status badge 2026-10-12 02:32:41 +08:00
fb43dd9716 fix(managed): honor metadata ExtraData before table decoding (#1893)
Co-authored-by: LouisChen <3361008+chqcjq@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 02:28:22 +08:00
0485cd3bce fix(javascript): preserve graphs within parent capacity (#1921)
Reserve the next sequential static checkpoint using its protocol size and
existing retained-expansion accounting instead of 80% of the parent budget.
Preserve total/live heap protections and explain parent admission failures
with effective capacity, retention estimates and usable retained facts.

Fixes #1911

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-12 02:17:38 +08:00
pablodiazjorgeandClaude Opus 5.5 c32d84622f test(clients): verify native Claude Code integration on Windows (#1926)
The Claude Code client lane asserted POSIX, although nothing in it depends
on it. Remove the assertion, tell Windows users to select claude.exe when
the default command cannot be spawned (npm installs put only a claude.cmd
shim on PATH), and record 2.1.296 as verified on native Windows in
docs/testing.md.

Fixes #1923

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 02:14:55 +08:00
morluto 601814ed0e fix(hopper): preserve observed process failures across requests (#1927)
* fix(hopper): preserve observed process failures across requests

* test(hopper): consolidate retained cleanup and recover failed helper fixtures

* docs(hopper): clarify first-use demo prompt and startup recovery
2026-10-12 02:13:53 +08:00
dfe31172ca fix(test): audit static Wasm and Harmony tool wording (#1917)
Account for explicit nonexecution wording in the existing tool-effects audit.
WABT validates and decompiles the module; Harmony projects inventory Evidence.
Retain declared effects and stale-exception checks.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 01:49:33 +08:00
morluto 6407718f30 test(managed): use the current attribute fixture contract 2026-10-12 01:42:02 +08:00
morluto f8513a42dc fix(javascript): keep TypeScript substitution out of runtime paths 2026-10-12 01:41:23 +08:00
LouisChenandLouisChen a95795c6ed fix(managed): resolve member ownership through pointer tables (#1886)
Co-authored-by: LouisChen <3361008+chqcjq@users.noreply.github.com>
2026-10-12 01:40:48 +08:00
morluto 369cd3dfdc fix(javascript): normalize package URLs before array selection (#1916) 2026-10-12 01:36:42 +08:00
morluto 2976fac30d test(architecture): consolidate import boundary verification 2026-10-12 01:31:33 +08:00
morluto f598bab488 fix(apple): preserve ARM64E X1 architecture identity 2026-10-12 01:31:33 +08:00
morluto bb508134ef fix(setup): verify independently selected skill destinations 2026-10-12 01:31:32 +08:00
morluto 1dfb933e47 fix(javascript): share TypeScript emitted import resolution 2026-10-12 01:31:32 +08:00
morluto 486b56724c fix(browser): compare only observed HTTP response facts 2026-10-12 01:31:23 +08:00
9d8e42bdd6 fix(javascript): follow actual receivers in nested intrinsic calls (#1912)
Consume arguments per generic function invocation rather than property-chain
depth. Preserve the callable receiver through call/apply, Reflect.apply and
bound invocation. Cover nested mutations and fresh targets through source
analysis and inert production CLI/MCP fixtures.

Fixes #1909.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
2026-10-12 01:22:26 +08:00
DeryFerd 499ac18011 fix(web): reject NUL bytes in web tool local path inputs
Four web tools accepted caller-selected local paths as plain strings, so
a NUL byte passed contract validation and reached filesystem calls. The
services only checked isAbsolute, and isAbsolute admits a NUL, so the
failure surfaced later as a raw ERR_INVALID_ARG_VALUE instead of a named
input constraint. Every other local-input tool already refuses NUL through
the shared local-path schema, and the existing OS-invalid-local-input
harness covers those but not these web tools.

inspect_web_network_capture.capture_path, export_web_scripts.capture_path
and output_directory, trace_web_source_location.manifest_path and
source_map.path, and trace_web_module_imports.manifest_path and
import_map.path now use localPathStringSchema. Each field already required
a nonempty string, so the advertised pattern and minLength stay the same
and only the NUL refusal is added.

Validation: a new schema test rejects NUL in all seven fields while
accepting ordinary absolute paths; the four tools are added to the
OS-invalid-local-input harness and reject NUL through the real advertised
MCP contracts; typecheck, oxlint, oxfmt, and toolSchemaValidity with a
regenerated catalog. Two webScriptExport boundary cases fail identically
with and without this change on this Windows host (symlink EPERM and
private-runtime-root authority), so they are environmental.
2026-10-11 23:32:52 +07:00
345 changed files with 47463 additions and 35224 deletions

No files matched your search

@@ -9,7 +9,19 @@ provenance; inputs and recovered code are never executed. Recovery requires a
caller-supplied Wakaru on Linux x64; see [the recovery guide](https://github.com/morluto/rea/blob/main/docs/javascript-recovery.md).
Use `analyze_javascript_application` directly on the operator-supplied ASAR or
extracted tree. Exact `#alias` specifiers resolve through the importer's
extracted tree. When the question concerns known source paths and the connected
schema advertises `source_paths`, select exact inventoried paths such as
`["package.json", "main.js", "preload.js"]`. Include supporting manifests or
source maps explicitly; dependencies are not implicitly analyzed. REA retains
the complete artifact inventory and identity but reads source text and analyzes
only selected files. The result and summary record `source_selection`; files
outside scope and their relationships remain unknown, so partial application
coverage does not mean the selected source failed. Use repeated `--source-path`
on either static application CLI route. Omit the selection when the question
requires complete application analysis. `detail: "summary"` reduces delivery
and does not by itself reduce analysis work.
Exact `#alias` specifiers resolve through the importer's
enclosing package `imports` map to package-local files; nested `node_modules`
packages use their own map. Wildcard, recursive, and external-package import
targets stay unresolved with that limitation. When size is unknown or large and the connected schema advertises
+1 -1
View File
@@ -493,7 +493,7 @@ jobs:
tests/acceptance/webSourceLocationCli.test.ts
tests/boundary/cli/referenceSourceImportStatus.test.ts
tests/boundary/filesystem/asarInventoryFreshness.test.ts
tests/composition/javascript-applications/javascriptArtifactReconstruction.test.ts
tests/boundary/filesystem/javascript-applications/javascriptArtifactReconstruction.test.ts
tests/conformance/providers/ghidra/windowsCapabilities.test.ts
src/ghidra/GhidraInstallation.test.ts
tests/boundary/providers/ghidra/ghidraTransport.test.ts
+2 -2
View File
@@ -1,6 +1,6 @@
{
"_comment": "Test files are intentionally NOT excluded: the gate used to report zero clones because it skipped **/*.test.ts, hiding all test-tree duplication. Current actual is ~2.47% lines across src+ and tests/, down from 3.4% when tests were included. The 2.6% limit leaves modest headroom for ordinary changes; lower it as duplication is removed.",
"threshold": 2.6,
"_comment": "Test files remain included so the gate measures duplication across both src/ and tests/. The 3.0% limit leaves headroom for ordinary changes while keeping duplication visible.",
"threshold": 3.0,
"reporters": ["consoleFull"],
"ignore": [
"node_modules/**",
+2
View File
@@ -18,6 +18,8 @@ Give each interpretation and resource a clear owner. Preserve identity, omission
Assess brittleness in touched code during implementation, debugging, review, and maintenance. Fix the producing representation or ownership rule before adding downstream repairs. Normalize once at the owning boundary using format-aware parsers. Remove superseded representations, aliases, wrappers, and redundant tests after verifying affected consumers. Introduce abstractions only when they remove observed complexity.
When fixing a bug, inspect the producing boundary and affected consumers for duplicated interpretation or unclear ownership. Complete any directly related refactor in the same change and remove superseded code. Passing regression tests alone does not establish that the brittleness is resolved. If related cleanup remains, explain what remains and why.
Parse unknown boundary values and model expected failures with the tagged error algebra and `Result`. Preserve partial facts and meaningful failure reasons. Keep identity and lookup values separate from display formatting; preserve source values and explicit unknowns when normalization loses information. Absolute paths, file URLs, and HTTP paths have different semantics.
Validate advertised JSON Schemas after SDK conversion against their declared dialect and actual clients. Bind handlers to named contracts with exact types; catalog ordering must never select an operation or schema.
+3 -2
View File
@@ -9,7 +9,6 @@
**See a feature you like. Understand how it works, down to the binary level.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#what-you-can-analyze)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
@@ -245,7 +244,9 @@ Ghidra and IDA use your existing installations. See [provider setup](docs/instal
<summary><strong>Do I need to start Hopper first?</strong></summary>
REA starts Hopper when an operation needs it. On macOS, a first-run dialog may
ask you to choose demo mode or activate your license. See
ask you to choose demo mode or activate your license. Complete that prompt in the
same user session before your first CLI/MCP operation or unattended job. Open
Hopper manually if needed. See
[Hopper startup and troubleshooting](docs/installation.md#launcher-paths-and-troubleshooting).
</details>
-1
View File
@@ -9,7 +9,6 @@
**وجدت ميزة تعجبك؟ افهم طريقة عملها حتى مستوى الشيفرة الثنائية.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Eine Funktion gefällt dir? Verstehe, wie sie funktioniert – bis hinunter auf die Binärebene.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**¿Ves una función que te gusta? Entiende cómo funciona, hasta el nivel binario.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**قابلیتی دیده‌اید که دوستش دارید؟ سازوکارش را تا سطح باینری درک کنید.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Repérez une fonctionnalité qui vous plaît. Comprenez son fonctionnement, jusqu'au niveau du binaire.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Melihat fitur yang Anda sukai. Pahami cara kerjanya hingga ke tingkat biner.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Vedi una funzionalità che ti piace. Capisci come funziona, fino al livello binario.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#cosa-puoi-analizzare)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**気に入った機能を見つけたら、その仕組みをバイナリのレベルまで理解する。**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**마음에 드는 기능을 발견했다면, 바이너리 수준까지 작동 원리를 이해하세요.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Widzisz funkcję, która Ci się podoba? Dowiedz się, jak działa, aż do poziomu kodu binarnego.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Viu uma funcionalidade de que gostou? Entenda como ela funciona, até o nível binário.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Нашли интересную функцию? Разберитесь, как она работает, вплоть до двоичного кода.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**เห็นฟีเจอร์ที่ชอบแล้วอยากรู้ว่าทำงานอย่างไร? เข้าใจได้ลึกถึงระดับไบนารี**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Beğendiğiniz bir özellik mi gördünüz? İkili kod düzeyine kadar nasıl çalıştığını anlayın.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Побачили цікаву функцію? Зрозумійте, як вона працює, аж до рівня двійкового коду.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**Thấy một tính năng bạn thích? Hiểu cách nó hoạt động, đến tận mức mã nhị phân.**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**看到喜歡的功能,弄清它的工作原理,深入到二進位層面。**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
-1
View File
@@ -9,7 +9,6 @@
**看到喜欢的功能,弄清它的工作原理,深入到二进制层面。**
[![npm version](https://img.shields.io/npm/v/rea-agents?style=flat-square&color=cb3837)](https://www.npmjs.com/package/rea-agents)
[![CI](https://img.shields.io/github/actions/workflow/status/morluto/rea/ci.yml?branch=main&style=flat-square&label=CI)](https://github.com/morluto/rea/actions/workflows/ci.yml)
[![MCP tool catalog](https://img.shields.io/badge/MCP-tool_catalog-5c4ee5?style=flat-square)](docs/mcp-contracts.md#generated-catalog)
[![Node.js requirements](https://img.shields.io/badge/Node.js-requirements-339933?style=flat-square&logo=nodedotjs&logoColor=white)](#current-status)
[![skills.sh](https://skills.sh/b/morluto/rea?style=flat-square)](https://skills.sh/morluto/rea/reverse-engineer-anything)
@@ -0,0 +1,38 @@
import java.util.Map;
import ghidra.app.plugin.core.analysis.AutoAnalysisManager;
import ghidra.app.util.headless.HeadlessScript;
/** Disable the named auto-analyzers before default auto-analysis. */
public final class ReaGhidraAnalysisOptions extends HeadlessScript {
@Override
public void run() throws Exception {
String[] arguments = getScriptArgs();
if (arguments.length != 1 || arguments[0].isBlank()) {
throw new IllegalArgumentException(
"ReaGhidraAnalysisOptions requires one comma-separated analyzer-name argument");
}
String[] names = arguments[0].split(",");
AutoAnalysisManager analysis = AutoAnalysisManager.getAnalysisManager(currentProgram);
Map<String, String> options = getCurrentAnalysisOptionsAndValues(currentProgram);
for (String requestedName : names) {
monitor.checkCancelled();
String name = requestedName.trim();
if (analysis.getAnalyzer(name) == null || !options.containsKey(name)) {
throw new IllegalArgumentException("Unknown Ghidra analyzer for this program: " + name);
}
}
for (String requestedName : names) {
monitor.checkCancelled();
setAnalysisOption(currentProgram, requestedName.trim(), "false");
}
options = getCurrentAnalysisOptionsAndValues(currentProgram);
for (String requestedName : names) {
monitor.checkCancelled();
String name = requestedName.trim();
if (!"false".equals(options.get(name))) {
throw new IllegalStateException("Ghidra analyzer remained enabled: " + name);
}
}
}
}
+5
View File
@@ -450,6 +450,11 @@ public final class ReaGhidraBridge extends HeadlessScript {
String seeds = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA analysis seeds", null);
if (seeds != null) result.add("analysis_seeds", JsonParser.parseString(seeds));
// Written by ReaGhidraMipsSoftFloat after analysis; REA requires it
// exactly when the committed profile declares soft-float storage.
String softFloat = currentProgram.getOptions(ghidra.program.model.listing.Program.PROGRAM_INFO)
.getString("REA MIPS soft-float storage", null);
if (softFloat != null) result.add("mips_soft_float_storage", JsonParser.parseString(softFloat));
return result;
}
@@ -0,0 +1,14 @@
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
// Compile the no-op activator with REA's scripts. Otherwise Ghidra generates
// one afterward and opens a second Java compiler and file manager.
public final class ReaGhidraBundleActivator implements BundleActivator {
@Override
public void start(BundleContext context) {
}
@Override
public void stop(BundleContext context) {
}
}
+176
View File
@@ -0,0 +1,176 @@
import ghidra.app.script.GhidraScript;
import ghidra.program.database.SpecExtension;
import ghidra.program.model.data.AbstractFloatDataType;
import ghidra.program.model.data.DataType;
import ghidra.program.model.data.DoubleDataType;
import ghidra.program.model.data.FloatDataType;
import ghidra.program.model.data.FunctionDefinition;
import ghidra.program.model.data.IntegerDataType;
import ghidra.program.model.data.ParameterDefinition;
import ghidra.program.model.data.TypeDef;
import ghidra.program.model.lang.CompilerSpec;
import ghidra.program.model.lang.PrototypeModel;
import ghidra.program.model.lang.Register;
import ghidra.program.model.listing.Function;
import ghidra.program.model.listing.Parameter;
import ghidra.program.model.listing.Program;
import ghidra.program.model.listing.VariableStorage;
import ghidra.program.model.pcode.XmlEncode;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.regex.Pattern;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
/**
* Soft-float o32 passes and returns float/double values exactly where same-size integers go.
* Ghidra's MIPS32 compiler specifications model only hard-float o32, so every typed float
* prototype (functions, externals, thunks via their targets, and function-pointer definitions)
* would otherwise read $f12/$f14/$f0. Derive a program-specific prototype model from the
* program's own default o32 model with its floating-point entries and rules removed, install it
* as a specification extension, and assign it to every callable whose formal types use floats.
*/
public final class ReaGhidraMipsSoftFloat extends GhidraScript {
static final String REPORT_KEY = "REA MIPS soft-float storage";
static final String POLICY = "o32-soft-float-prototype-v1";
static final String PROTOTYPE = "__rea_soft_float";
private static final Pattern FPU_REGISTER = Pattern.compile("f\\d+(_\\d+)?");
@Override
public void run() throws Exception {
String processor = currentProgram.getLanguage().getProcessor().toString();
if (!processor.equals("MIPS") || currentProgram.getDefaultPointerSize() != 4)
throw new IllegalStateException(
"REA soft-float storage requires a 32-bit MIPS program, not " + processor);
CompilerSpec spec = currentProgram.getCompilerSpec();
PrototypeModel base = spec.getDefaultCallingConvention();
new SpecExtension(currentProgram)
.addReplaceCompilerSpecExtension(softFloatPrototype(base, spec), monitor);
PrototypeModel soft = currentProgram.getCompilerSpec().getCallingConvention(PROTOTYPE);
if (soft == null)
throw new IllegalStateException("REA soft-float prototype was not installed");
requireIntegerStorage(
soft.getStorageLocations(
currentProgram,
new DataType[] {
FloatDataType.dataType, IntegerDataType.dataType, DoubleDataType.dataType,
FloatDataType.dataType, DoubleDataType.dataType
},
false),
"soft-float prototype self-check");
int functions = 0;
List<Function> callables = new ArrayList<>();
currentProgram.getFunctionManager().getFunctions(true).forEach(callables::add);
currentProgram.getFunctionManager().getExternalFunctions().forEach(callables::add);
for (Function function : callables) {
// A thunk's signature is its target's; the target is in this list.
if (function.isThunk() || !usesFloat(function)) continue;
if (function.hasCustomVariableStorage()) function.setCustomVariableStorage(false);
function.setCallingConvention(PROTOTYPE);
List<VariableStorage> storage = new ArrayList<>();
storage.add(function.getReturn().getVariableStorage());
for (Parameter parameter : function.getParameters()) storage.add(parameter.getVariableStorage());
requireIntegerStorage(storage.toArray(VariableStorage[]::new), function.getName(true));
functions += 1;
}
int definitions = 0;
Iterator<DataType> types = currentProgram.getDataTypeManager().getAllDataTypes();
while (types.hasNext()) {
if (!(types.next() instanceof FunctionDefinition definition) || !usesFloat(definition))
continue;
definition.setCallingConvention(PROTOTYPE);
definitions += 1;
}
// Written last: any failure above leaves no report, and REA refuses the session.
currentProgram.getOptions(Program.PROGRAM_INFO).setString(
REPORT_KEY,
"{\"policy\":\"" + POLICY + "\",\"functions\":" + functions +
",\"definitions\":" + definitions + "}");
println("REA MIPS soft-float prototype: functions=" + functions + " definitions=" + definitions);
}
/** The default o32 model without floating-point registers or float-consuming rules. */
private static String softFloatPrototype(PrototypeModel base, CompilerSpec spec) throws Exception {
XmlEncode encoder = new XmlEncode();
base.encode(encoder, spec.getPcodeInjectLibrary());
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/disallow-doctype-decl", true);
Document document =
factory.newDocumentBuilder().parse(new InputSource(new StringReader(encoder.toString())));
Element root = document.getDocumentElement();
if (!root.getTagName().equals("prototype"))
throw new IllegalStateException("Unexpected default prototype encoding " + root.getTagName());
root.setAttribute("name", PROTOTYPE);
List<Node> removed = new ArrayList<>();
collect(root.getElementsByTagName("pentry"), "metatype", "float", null, removed);
collect(root.getElementsByTagName("consume"), "storage", "float", "rule", removed);
if (removed.isEmpty())
throw new IllegalStateException("Default prototype has no floating-point entries to remove");
for (Node node : removed) node.getParentNode().removeChild(node);
StringWriter xml = new StringWriter();
var transformer = TransformerFactory.newInstance().newTransformer();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.transform(new DOMSource(document), new StreamResult(xml));
return xml.toString();
}
/** Select matching elements, or their nearest `ancestor` element when one is named. */
private static void collect(
NodeList nodes, String attribute, String value, String ancestor, List<Node> selected) {
for (int index = 0; index < nodes.getLength(); index += 1) {
Element element = (Element) nodes.item(index);
if (!value.equals(element.getAttribute(attribute))) continue;
Node target = element;
while (ancestor != null && target != null && !ancestor.equals(target.getNodeName()))
target = target.getParentNode();
if (target != null && !selected.contains(target)) selected.add(target);
}
}
private static void requireIntegerStorage(VariableStorage[] storage, String owner) {
for (VariableStorage item : storage) {
if (item.isBadStorage() || item.isUnassignedStorage())
throw new IllegalStateException("REA soft-float storage unresolved for " + owner);
List<Register> registers = item.getRegisters();
if (registers == null) continue; // stack-only storage
for (Register register : registers) {
if (FPU_REGISTER.matcher(register.getName()).matches())
throw new IllegalStateException(
"REA soft-float storage still uses " + register.getName() + " for " + owner);
}
}
}
private static boolean usesFloat(Function function) {
if (isFloat(function.getReturn().getFormalDataType())) return true;
for (Parameter parameter : function.getParameters()) {
if (isFloat(parameter.getFormalDataType())) return true;
}
return false;
}
private static boolean usesFloat(FunctionDefinition definition) {
if (isFloat(definition.getReturnType())) return true;
for (ParameterDefinition argument : definition.getArguments()) {
if (isFloat(argument.getDataType())) return true;
}
return false;
}
private static boolean isFloat(DataType type) {
DataType current = type;
while (current instanceof TypeDef typeDef) current = typeDef.getBaseDataType();
return current instanceof AbstractFloatDataType;
}
}
+7 -1
View File
@@ -108,7 +108,13 @@ existing analysis families directly, for example
`ro.kernel.qemu` value, `null` when absent.
- **Device paths are not host paths.** `base_apk_device_path` values live on
the device filesystem. Only pulls and captures produce host-local paths,
each paired with byte size and SHA-256 digest.
each paired with byte size and SHA-256 digest. A pulled file's host name is
the device path's last segment, accepted only when every host can store it:
a `pull_adb_file` whose device path ends in a separator, `.`/`..`, a
control or reserved character, a Windows reserved device name, a trailing
dot or space, or a `:` alternate-data-stream marker is refused as
`invalid_request` before any `adb pull` runs, and REA never renames a
device-reported file.
- **Splits are part of the answer.** `pm path` enumerates the base APK and
every installed split; pulls cover the whole set. System packages whose
APKs keep other names report the `unknown` role rather than guessing.
+99 -15
View File
@@ -13,13 +13,13 @@ retains class, byte order, `e_type`, raw `e_flags`, and inspected ABI declaratio
Recognition does not mean every provider or MIPS variant is supported. Existing
PE, managed and Mach-O architecture sets are unchanged.
The first Ghidra lane admits ELF32 `ET_EXEC`, little or big endian, with explicit
o32 and `EF_MIPS_ARCH_32R2` flags and no machine-specific or ASE encoding flags.
These are header declarations, not proof that every instruction conforms to
that ISA. ELF64, n32/n64, unspecified ABI, MIPS16/microMIPS, machine-specific
variants, shared/relocatable objects and PSP PRX are not covered by this lane.
Their family may be identified, but provider selection returns the specific
unsupported constraint rather than guessing a compatible profile.
The Ghidra lane admits ELF32 `ET_EXEC` and `ET_REL`, little or big endian, with
explicit o32 and `EF_MIPS_ARCH_32R2` flags and no machine-specific or ASE
encoding flags. These are header declarations, not proof that every instruction
conforms to that ISA. ELF64, n32/n64, unspecified ABI, MIPS16/microMIPS,
machine-specific variants, shared objects and PSP PRX are not covered by this
lane. Their family may be identified, but provider selection returns the
specific unsupported constraint rather than guessing a compatible profile.
Explicit `EF_MIPS_FP64` and `EF_MIPS_NAN2008` declarations are refused. Their
absence is insufficient: Clang can emit identical `e_flags` for FP32, FP64 and
@@ -39,12 +39,14 @@ absent or supply a default ABI.
The field layout follows
[`Elf_MIPS_ABIFlags_v0`](https://github.com/bminor/glibc/blob/master/elf/elf.h).
Provider admission requires record version 0, MIPS32r2, 32-bit GPR/CPR1, no CPR2,
and a double-precision or FPXX ABI declaration with no ISA extension, ASE or
unknown general flags. Missing records remain unknown rather than defaulting
to a compatible ABI, and this initial lane refuses them. Unknown record values
are retained for diagnosis but do not gain provider support. These checks bind
the declared interpretation; the integer-only fixture does not establish full
Provider admission requires record version 0, MIPS32r2, 32-bit GPRs, no CPR2,
no ISA extension, ASE or unknown general flags, and one of two floating-point
declarations: 32-bit CPR1 with a double-precision or FPXX ABI, or no CPR1 with
the soft-float ABI. Other combinations, including single-float, FP64 and
unspecified ABIs, are refused. Missing records remain unknown rather than
defaulting to a compatible ABI, and this lane refuses them. Unknown record
values are retained for diagnosis but do not gain provider support. These checks
bind the declared interpretation; the fixtures do not establish full
floating-point behavioral conformance.
MIPS header and ABI record facts are committed in the analysis profile so
@@ -53,6 +55,51 @@ identity is versioned independently of file identity. Evidence, lifecycle
outputs and saved snapshots preserve the `mips` family. Hopper and IDA adapters
do not implicitly gain MIPS support. Windows Ghidra P0 remains PE x86/x86-64 only.
### Soft-float prototype
Ghidra's MIPS32 compiler specifications model hard-float o32 only: a typed
`float` or `double` parameter is bound to `$f12`/`$f14` and returned in `$f0`.
Soft-float o32 passes those values in integer registers and stack slots, so a
typed prototype (for example from DWARF) would make the decompiler read FPU
registers the code never writes and drop the real dataflow. That applies to
functions, external functions and indirect calls through typed function
pointers alike.
For soft-float targets REA commits
`mips_soft_float_storage: o32-soft-float-prototype-v1` in the analysis profile
and runs `ReaGhidraMipsSoftFloat.java` after auto-analysis. The script encodes
the program's own default o32 prototype, removes its floating-point parameter
and return entries and the rules that consume floating-point storage, and
installs the result as the program-specific `__rea_soft_float` prototype through
a Ghidra specification extension. Deriving it keeps the byte-order-specific
aggregate rules and the hidden struct-return pointer. The script then assigns
that prototype, with dynamic storage, to every non-thunk function and external
function, and to every function-pointer definition, whose declared types use
`float` or `double`. Thunks follow their targets. Types are kept.
The script fails, and writes no report, if the derived prototype still places a
float in an FPU register or any reassigned function does. It records the
function and definition counts last; REA requires that report exactly when the
profile commits the policy and refuses the session otherwise, before serving any
operation. Session Evidence carries the policy and counts as limitations.
Callables without typed float signatures keep Ghidra's default prototype, under
which Ghidra already infers integer-register parameters. Hard-float profiles do
not gain the policy, so their digests are unchanged.
### Relocatable objects
`ET_REL` objects, such as Linux kernel modules (`.ko`), are imported by Ghidra's
ELF loader, which assigns each allocated section an analysis address and applies
the object's REL relocations there. Undefined symbols are placed in Ghidra's
`EXTERNAL` block: a thunk for code, a label for data. These addresses are
analysis coordinates, not the addresses a kernel or linker would choose, and
session Evidence says so. `inspect_native_load_image` reports each relocation
with Ghidra's status; HI16 entries report `PARTIAL` because Ghidra applies them
together with their paired LO16. Relocatable targets commit
`mips_support_lane: elf32-rel-o32-arch32r2-standard-v1`, separate from the
executable lane, whose identity executable targets keep. Kernel modules are
usually soft-float and therefore also use the soft-float storage policy.
## Use
With a caller-installed Ghidra and compatible JDK on a supported Linux/macOS host:
@@ -70,7 +117,10 @@ REA's ELF parser.
## Verification
The source-owned freestanding fixture is `tests/conformance/c/mips.c`. The
### Executables
The source-owned freestanding fixture is `tests/conformance/c/mips.c`; the
soft-float variant also links `tests/conformance/c/mips-soft-float.c`. The
optional cross-target lane requires a caller-supplied Clang with MIPS targets,
LLD, GNU `readelf`, Ghidra and its compatible JDK; ordinary host-native checks
do not acquire these tools. `REA_MIPS_CLANG` selects Clang and
@@ -81,7 +131,8 @@ These are test-time prerequisites, not REA runtime providers.
GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra npm run verify:ghidra:mips
```
The lane compiles both byte orders and never executes the targets. GNU readelf
The lane compiles both byte orders with hard-float and soft-float ABIs and never
executes the targets. GNU readelf
independently supplies header/ABI facts and symbol addresses, compared with the
production REA target resolver before provider startup. Its version, bounded
command output and selected tool path are retained in the verification report.
@@ -104,9 +155,42 @@ all program-header payloads. It checks the actual Ghidra language, entry-address
decompilation, distinct artifact digest and missing-section limitations through
CLI Evidence and an MCP-produced snapshot. Caller-selected Ghidra heap limits
are passed to these child workflows.
For soft-float fixtures the lane also checks, through `inspect_native_api`, the
exact parameter and return types and storage of float, double, mixed (pair
alignment and stack spill) and struct-returning functions; that a caller's
decompilation receives every source float constant without FPU inputs,
including an indirect call through a typed function pointer; and that the
snapshot profile and Evidence retain the policy and outcome counts.
Pseudocode presence is a liveness check, not proof of semantic equivalence.
Instruction decoding and reader agreement likewise do not establish runtime,
delay-slot execution or complete floating-point semantics. Assertion regressions
reject incorrect byte order, constants, destinations and loaded identities.
Parser/profile tests are narrower checks, not substitutes for this real-provider
run. Report the actual lanes run and leave unexecuted verification pending.
### Relocatable objects
This lane has its own command and fixture:
```bash
GHIDRA_INSTALL_DIR=/absolute/path/to/ghidra npm run verify:ghidra:mips-relocatable
```
It compiles `tests/conformance/c/mips-relocatable.c` with `-c -G0` in both byte
orders, plus a little-endian soft-float `-mlong-calls` variant matching Linux
module options. GNU readelf independently supplies sections, symbols and REL
entries. For every relocation in an allocated section, the lane reads the
loaded word through `read_bytes` and requires it to encode `S + A`: `S` from
Ghidra's own section placement (`list_segments`) or `EXTERNAL` placement, `A`
from the original file bytes. HI16/LO16 are checked as the effective address
each LO16 forms with each HI16 it resolves, including several HI16s before one
LO16, so a missing signed-low carry fails and a HI16 without a following LO16
fails. The lane also requires matching Ghidra relocation entries and statuses,
section file offsets and sizes, coverage of a shared HI16, a LO16 carry,
external code and data and `R_MIPS_32` words, direct calls from the entry
function, the relocatable support lane in the snapshot profile, the
relocatable limitation in Evidence and an unchanged source object. The fixture
also calls `powf`, an external whose float prototype comes from Ghidra's C
library archive; every variant must decompile `powf(1.5,2.5)` without FPU
inputs, and the soft-float variant must report that the soft-float prototype
covered it.
+1 -1
View File
@@ -98,7 +98,7 @@ For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client <cl
## Coverage and evidence
Both profiles support procedure/string inventory and literal or regex search, procedure resolution, pseudocode, assembly, raw function instructions, resolved callees, incoming address references, and a function dossier. Inventories and modern pseudocode/assembly follow upstream pagination. Upstream limits on modern xrefs and callees are retained as explicit truncation limitations. A transport-level truncated preview is rejected rather than reported as complete evidence; the error preserves upstream retrieval guidance.
Both profiles support procedure/string inventory and literal or regex search, procedure resolution, pseudocode, assembly, raw function instructions, resolved callees, incoming address references, and a function dossier. Inventories and modern pseudocode/assembly follow upstream pagination. Upstream limits on modern xrefs and callees are retained as explicit truncation limitations. In the modern headless profile, targets that are not proven x86-only use processor-neutral `callgraph` edges instead of the x86 instruction checks in upstream `callees` (mrexodia/ida-pro-mcp#549). REA requests depth zero: the pinned upstream emits the root's outgoing edges without scanning child functions, then REA batches function-name lookup for those addresses. The graph covers recognized function destinations only; imports and other non-function targets remain unknown. A server without `callgraph` returns an empty relationship list with an explicit coverage limitation. Graph and path workflows retain those limitations alongside observed edges. A failed or inconsistent graph reply remains an error. A transport-level truncated preview is rejected rather than reported as complete evidence; the error preserves upstream retrieval guidance.
The legacy profile additionally supports direct callers. Its producer may report call-site or interior addresses; REA resolves the reported function symbol to its canonical entry. The modern profile does not prove direct callers from its code/data xref labels, so that capability is unavailable and the dossier marks it unknown.
+39 -2
View File
@@ -352,6 +352,12 @@ Other integrations, including Qwen Code, use `~/.agents/skills`.
A mixed selection plans each distinct destination. Selecting the skill without a client uses the shared
directory. Setup leaves existing skill copies in other locations untouched.
`rea setup --skill-client claude_code --yes --json` refreshes and verifies the
selected skill destination independently of MCP registration selection. The
final `doctor.scope.skill_clients` records which skill destinations were
checked; `doctor.scope.clients` continues to describe required registrations.
An unselected shared skill does not affect a client-only refresh.
`doctor --skill --json` verifies the selected copies against the bundled
instructions and references. Consumers should use `identity.skill.state`,
`installed_version`, and `installed_tool_count` for skill readiness. The
@@ -561,8 +567,13 @@ need Xvfb, Python 3, X11 and XTEST; approved setup installs those dependencies
on supported distributions. Add the curl installer's reported executable
directory, commonly `~/.local/bin`, to your shell `PATH` if needed.
REA starts Hopper when needed. On macOS, its launcher can bring a window or
first-run dialog forward; choose demo mode or activate your existing license.
REA starts Hopper when needed. On macOS, complete its demo or license prompt in
the same user session before the first CLI/MCP operation or unattended job. Open
Hopper manually if needed; its launcher can also bring the dialog forward.
If startup reports `Cannot launch Hopper! Connection is invalid.` or waits for
bridge readiness, check Hopper for a pending demo, license, or loader dialog.
Complete the dialog and retry. If REA reports `cleanup_incomplete`, follow the
retained-resource guidance below before reopening the target.
Hopper serializes analysis requests. Cancelling a wait can leave provider work
running, which the session reports. Successful decompilation is cached until
a relevant rename or comment changes it.
@@ -591,6 +602,11 @@ if document closure is unconfirmed, REA retains the backing image and reports
MCP-exit behavior.
Startup deadlines report missing bridge readiness and preserve the launcher
outcome; a successful helper exit does not prove that a loader dialog completed.
If an owned provider process exits, request failures preserve its observed exit
code or signal and bounded output in `details.launcher`. An exit can precede
output-stream closure; `output_closed: false` keeps that uncertainty explicit.
Later requests retain the known exit instead of reducing it to socket loss.
An external GUI's helper exit does not establish that the GUI itself exited.
Cursor navigation returns the observed object start when Hopper snaps an interior
address; adjacent-object navigation rejects unmapped inputs and document ends.
Native API text rejects NUL characters and unpaired Unicode surrogates before
@@ -784,6 +800,27 @@ Each line is `address<TAB>kind[<TAB>name]`, where kind is `function`, `code` or
Label seeds can name mapped uninitialized storage such as `.bss`; function and code seeds require initialized bytes.
To skip a named Ghidra auto-analyzer (for example, when default auto-analysis
stalls in one analyzer on a Swift/Objective-C arm64 Mach-O):
```bash
export REA_GHIDRA_DISABLED_ANALYZERS="Objective-C Message Analyzer"
```
The value is a comma-separated list of exact Ghidra analyzer names. A pre-script
disables each name before auto-analysis, the analysis profile records the
override in `analyzer_preset` (for example
`ghidra-default+disabled:Objective-C Message Analyzer`), and a session fails to
open when its committed profile and the setting differ. A name starts with a
letter or digit and contains only letters, digits, spaces, periods, underscores,
and hyphens; at most 32 names are admitted, each at most 120 characters.
Names must identify analyzers registered for the selected program. Unknown names
or options that remain enabled abort the headless import before auto-analysis
and the bridge, so an unsuccessful override cannot produce a session claiming
that the analyzer was disabled. For Windows PE resource analysis, the exact name
is `WindowsResourceReference`. Disabling an analyzer skips the facts it would
have produced.
### Ghidra heap and CPU controls
Set resource controls in the environment that launches the CLI or MCP server,
+77 -12
View File
@@ -66,6 +66,45 @@ code-point order. Dependency manifests remain package observations but do not
declare Electron main or renderer entries. If only dependency manifests are
present, the inventory artifact remains the graph root.
## Select sources before analysis
For a question about known entry sources, supply exact artifact-relative file
paths in `source_paths`, or repeat `--source-path` on either CLI route:
```bash
rea analyze /absolute/path/to/app.asar \
--source-path package.json --source-path main.js --json
```
```json
{
"input_path": "/absolute/path/to/app.asar",
"source_paths": ["package.json", "main.js"],
"detail": "summary"
}
```
REA inventories and verifies the complete artifact, including unselected bytes,
but allocates source text and performs static analysis only for selected files.
Include supporting manifests, HTML or source maps explicitly when needed;
imports do not expand the selection. Paths must match relevant inventoried
files exactly, including nested ASAR prefixes. Unknown paths, aliases and globs
return an `invalid_request` correction. Selected files are analyzed in request
order before unselected scopes are projected. Repeated paths preserve the
caller's request and first-occurrence priority, analyzing the file once.
Omit the selection for the existing complete
application workflow.
The result and summary retain `source_selection.requested_paths` and
`source_selection.unselected_files`. Every relevant file keeps its original
inventory identity; unselected files report `text_status:
"outside-analysis-scope"`. Their relationships remain unknown, and application
coverage is partial while any file remains outside scope. Selection does not
claim whole-application completeness or change integrity policy. Summary detail
only reduces delivery; source selection reduces text acquisition and analysis.
Explicit native provider or snapshot routing rejects `--source-path` rather
than ignoring it.
## ASAR integrity
ASAR inventory checks Electron integrity metadata for embedded archive entries
@@ -159,9 +198,11 @@ completed partial Evidence. MCP failure details return
after recording succeeds, so `inspect_analysis_view` can open that analysis
directly without serializing the entire partial graph into the error response.
CLI failure output retains the complete partial Evidence.
Cancellation after source analysis begins likewise
retains completed facts; an SDK client may reject its cancelled request before
receiving that reply. On the same MCP connection, inspect the Evidence bundle
Cancellation after source analysis begins likewise retains completed facts,
including cancellation during analyzer cleanup, graph construction, or Evidence
sealing. REA finishes projecting those owned facts cooperatively to return partial
Evidence. An SDK client may reject its cancelled request before receiving that
reply. On the same MCP connection, inspect the Evidence bundle
and use the retained analysis ID with `inspect_analysis_view`. Owned analyzer
cleanup is verified before its private files are released. Cleanup uncertainty
is reported separately and retains ownership for retry.
@@ -231,6 +272,11 @@ unknown instead of silently treating them as absent or verified.
Plain `.ts` artifact sources use TypeScript syntax without JSX, including
angle-bracket type assertions. `.tsx` and `.jsx` retain JSX parsing.
Relative TypeScript imports can name their emitted `.js`, `.mjs`, or `.cjs`
counterparts while resolving to inventoried source files. This substitution
requires import or require semantics. Literal Electron preload, renderer, and
utility-process runtime paths retain the runtime filename selected by the caller.
Selected bounded text is then parsed as inert data to recover:
- `package.json` metadata and declared main or renderer entrypoints;
@@ -346,12 +392,14 @@ Known Object/Reflect mutators remain recognizable through literal destructuring
of intrinsic objects and stable bound-function aliases. Calls retain bound
argument prefixes and the original bound receiver through `call`, `apply`, and
`Reflect.apply`, including generic `Function.prototype` invocation and binding
creation. Visibly replaced invocation intrinsics produce unknown effects instead
creation. Arguments follow each actual callable invocation rather than property
chain depth; rebinding retains the original bound receiver.
Visibly replaced invocation intrinsics produce unknown effects instead
of using their former call semantics; captured function aliases retain their
earlier identity. Unexamined
targets remain unresolved. `Reflect.set` examines its explicit receiver as well
as its target, retaining unknown receiver effects. Creating an unused binding does not itself mutate
an intrinsic. Variable redeclarations retain a single unconditional initializer
earlier identity. Unexamined targets remain unresolved. `Reflect.set` examines
its explicit receiver as well as its target, retaining unknown receiver effects.
Creating an unused binding does not itself mutate an intrinsic. Variable
redeclarations retain a single unconditional initializer
when they introduce no assignment or second initializer.
Stable identifier chains to fresh object, array, function,
undecorated class or RegExp syntax keep those targets separate from intrinsic
@@ -460,6 +508,12 @@ from unmatched nested targets rather than reporting array indices.
For imports and requires, the active Node conditions include `module-sync`;
earlier active conditions retain precedence over later ones.
Package self-references use the importer's enclosing package scope before
installed dependencies. ESM requires the parsed package name to equal the
metadata name; a scope-only name such as `@scope` cannot capture `@scope/pkg`.
CommonJS retains Node's raw-request name/prefix matching. Both paths preserve
the original specifier and select exports under their own loader conditions.
Exact package-private `#alias` specifiers resolve against the importer's
enclosing `package.json` `imports` map — the nearest same-container package
scope, stopping at a `node_modules` directory so a nested package cannot inherit
@@ -475,13 +529,24 @@ URL targets before selecting a later package-local file; an unsupported bare
package or recursive alias keeps its unresolved outcome rather than selecting
a later fallback.
ESM relative module paths and selected package exports targets use URL suffix handling
and one percent-decoding pass. CommonJS relative paths and legacy package main
ESM relative module paths, legacy package subpaths without exports, and selected
package exports targets use URL suffix handling and one percent-decoding pass.
Bare specifiers retain their complete query/fragment text for exports key matching.
For dependency lookup, ESM constructs a package metadata URL instead of treating
the parsed package name as a literal directory. A root request such as `legacy?x`
does not load either `node_modules/legacy?x` or `node_modules/legacy`; an exact
self-reference with that metadata name can still select its exports first.
Without an exports map, ESM subpaths select exact files; directory imports and
extension fallback are refused. CommonJS relative paths and legacy package main
fields retain literal filename punctuation. Exports targets must start with
`./` and exclude `.`, `..`, and `node_modules` path segments, including their
encoded forms, before URL normalization. Arrays skip invalid exports targets
in declared order; URL decoding and file lookup failures for a selected target
do not select a later entry. Rejected references retain the package metadata
in declared order, including URLs that escape the package root after normalization.
Selected package targets use the normalized URL path for lookup; tabs and
newlines in the URL text are removed by the URL parser. CommonJS package targets
reject encoded separators anywhere in the selected URL; ESM checks its pathname.
URL decoding and file lookup failures for a selected target do not select a
later entry. Rejected references retain the package metadata
path, original target, failed constraint, and importing source location.
HTML script references resolve to exact inventoried files after applying the
+18
View File
@@ -126,6 +126,24 @@ Member inspection binds build-local tokens, signatures, row offsets, method
bodies, and decoded relationships to that artifact and MVID. Missing or malformed
facts remain explicit in coverage and per-member status.
Metadata table streams may carry the CLR-compatible `ExtraData` extension
(`HeapSizes` flag `0x40`), which Microsoft readers honor although it is not
specified by ECMA-335. REA skips its fixed four-byte word after the row counts
before decoding table rows; the word's value is not a length. An incomplete
word is reported as a metadata coverage failure at its file offset, without
reading beyond the declared table stream.
For unoptimized `#-` metadata, nonempty `FieldPtr` and `MethodPtr` tables
translate TypeDef list positions to physical member definitions. REA resolves
these indirections for member ownership and local attribute-constructor types;
field and method tokens still identify the original definition rows.
`field_list` and `method_list` retain the metadata's list coordinates, which
refer to the pointer table when present rather than necessarily contiguous
definition rows. Malformed or ambiguous pointers leave the owner unknown with
partial coverage while preserving readable member and attribute facts.
The retained inverse indexes are admitted against the existing representation
budget before allocation.
`exported_types` reports ExportedType rows as declared. Facade assemblies such
as .NET `System.Runtime.dll` provide their types this way: `forwarder: true`
marks a type forwarder, and `implementation_token` names the AssemblyRef, File,
+19 -2
View File
@@ -310,6 +310,13 @@ directly to `compare_functions`, and `inspect_artifact` Evidence to
`compare_artifacts`. Use `get_evidence_bundle` when the task needs broader
retained session history or an explicit bundle for transfer.
`get_call_graph` keeps provider-limited relationship reads on their node as
`limitations`, alongside any observed `calls`. An empty call list with such a
limitation is incomplete coverage, not proof that the function has no calls.
`trace_call_path` reports each limited expansion in `residual_unknowns` and
sets `truncated: true`; any path already supported by observed edges remains
visible. Tool errors remain in their existing error or failure fields.
REA prepares complete MCP results within the pinned stdio client's 10 MiB
receive-buffer budget, including both text and structured representations and
room for the JSON-RPC envelope. Tool errors carry only their text
@@ -321,8 +328,18 @@ session. Its exact reference is reported in
`details.reported_limits.evidence_reference`; call `inspect_analysis_view` with
that reference for a summary, one section/module, or a stable page, or call
`export_evidence_bundle` with a destination path. Complete
bundle exports stream canonical JSON into an atomically published file. A broad
follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
bundle exports stream canonical JSON into an atomically published file.
For oversized native function Evidence, `details.reported_limits.continuation`
contains a ready-to-call `inspect_analysis_view` request for the retained
pseudocode's first page. Use its `name` and `arguments` directly, then follow
`normalized_result.coverage.next_offset` for subsequent pages. The suggested
limit is the view contract's default, not a guarantee that arbitrary source or
metadata fits; reduce the limit or export when needed. Continuations are
offered for completed `analyze_function` results only after session recording is
acknowledged; the complete result and its retention reference remain intact.
A broad follow-up or `get_evidence_bundle` can also exceed the response budget; exporting
preserves the complete session without sending it through a single MCP frame.
To discover existing records without their large payloads, call
`get_evidence_bundle` with `detail: "summary"`. Its `result.records` contain
+11 -7
View File
@@ -35,14 +35,18 @@ Omitted `format` keeps header-based detection; architecture and entry without
`format=raw-binary` are rejected.
Ghidra imports a private snapshot with `BinaryLoader` and these little-endian
generic languages:
generic languages and compiler specifications:
| architecture | language_id |
| ------------ | ------------------- |
| x86 | `x86:LE:32:default` |
| x86_64 | `x86:LE:64:default` |
| arm | `ARM:LE:32:v7` |
| arm64 | `AARCH64:LE:64:v8A` |
| architecture | language_id | compiler_spec_id |
| ------------ | ------------------- | ---------------- |
| x86 | `x86:LE:32:default` | `gcc` |
| x86_64 | `x86:LE:64:default` | `gcc` |
| arm | `ARM:LE:32:v7` | `default` |
| arm64 | `AARCH64:LE:64:v8A` | `default` |
Compiler specifications select analysis conventions; they do not identify the
original compiler or ABI of a headerless target. Ghidra's x86 languages do not
provide a compiler specification named `default`.
Those settings, the caller-supplied entry, and `language_selection:
caller-supplied-v1` are committed in the analysis profile. A different entry
+20 -12
View File
@@ -139,8 +139,11 @@ directory, so default OS-home discovery remains unverified. Live model APIs and
native Windows are also unverified.
`verify:claude-client` requires installed Claude Code (verified with
`@anthropic-ai/claude-code@2.1.296`); select it with `REA_VERIFY_CLAUDE_COMMAND`.
The optional POSIX lane uses native `CLAUDE_CONFIG_DIR` discovery, an independent
`@anthropic-ai/claude-code@2.1.296` on POSIX and native Windows); select it
with `REA_VERIFY_CLAUDE_COMMAND`. On Windows, select
`claude.exe`: npm installs put only a `claude.cmd` shim on `PATH`, and the real
executable is under `node_modules/@anthropic-ai/claude-code/bin`.
The optional lane uses native `CLAUDE_CONFIG_DIR` discovery, an independent
Git workspace, user settings and default built-in tools. It preserves caller
preferences and backups, checks idempotent setup, waits through the native
`WaitForMcpServers` tool when discovery is pending, and validates the complete
@@ -149,8 +152,8 @@ native `Skill`, checks its full body and verifies named-schema JavaScript Eviden
for a Unicode path. Use `-- chat` for ordinary chat with REA enabled, or
`REA_VERIFY_RUNTIME_ROOT` for a production-only installed package. The local
Anthropic Messages/SSE fixture preserves native resource-hint envelopes in its
request artifacts. Usage is synthetic; live Anthropic API, native Windows and
bare-mode personal-skill activation remain unverified by this lane.
request artifacts. Usage is synthetic; live Anthropic API and bare-mode
personal-skill activation remain unverified by this lane.
`verify:copilot-client` requires installed GitHub Copilot CLI (verified with
`@github/copilot@1.0.95`); select it with `REA_VERIFY_COPILOT_COMMAND`.
@@ -279,10 +282,12 @@ establish live DeepSeek API or native Windows coverage. Receipts include
request artifacts, result digests and owned-process lineage.
Full E2E tests invoke the production command dispatcher and real providers,
without fake launchers, runners or responses. `verify:keyed-archive` writes an
actual Foundation binary archive, runs the CLI and a separate stdio MCP
subprocess, checks parity and pagination, rejects an escaping path and checks
an XML graph golden. `verify:asset-catalog` compiles source-owned colors with
without fake launchers, runners or responses. `verify:keyed-archive` writes
Foundation binary, XML, and integer archives using one run-owned Swift module
cache. It checks CLI and separate stdio MCP parity, pagination, escaping-path
rejection, an XML graph golden, and numeric meaning for safe controls, unsafe
integers, boxed numbers, and integral-valued reals. `verify:asset-catalog`
compiles source-owned colors with
`actool`, invokes real `assetutil`, then checks CLI/MCP results, exact catalog
digest, every raw metadata field, pagination and malformed input rejection.
Neither artifact workflow requires Hopper or launches it. Both run in macOS CI.
@@ -458,7 +463,7 @@ prove behavior on the verified Ghidra 12.1.4 and JDK 21 build.
| Ghidra lane | Supported runner/target | Additional local tools |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| `npm run verify:ghidra` | Linux x64/arm64 ELF or macOS x64/arm64 Mach-O | Host C compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX host; source-owned 32-bit x86 raw fixture; function order and BSS labels | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:seeds` | POSIX; raw x86 fixture; seed ordering, BSS labels, analyzer readback and unknown-name rejection | Ghidra 12.1.4 and full JDK 21 |
| `npm run verify:ghidra:swift` | macOS x64/arm64 Mach-O | Host Swift compiler, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:switch` | Linux x64 ELF; GCC/Clang optimized and stripped switch fixtures | GCC, Clang, GNU nm/objdump/strip, Ghidra 12.1.4, and full JDK 21 |
| `npm run verify:ghidra:aarch64-jump-table` | Any supported Ghidra host; AArch64 ELF; signed/unsigned byte/halfword tables; host ARM64 Mach-O | Clang with AArch64 target support, Ghidra 12.1.4, and full JDK 21 |
@@ -798,8 +803,10 @@ through the compiled CLI and stdio MCP server. With the same Ghidra/JDK
prerequisites, it checks preparation and bridge startup, verified load mappings,
instruction and byte readback, canonical-entry session reuse, replacement when
the entry changes without `provider_id`, unchanged source bytes, and owned
cleanup. Other raw CPU families remain covered by deterministic tests rather
than this real-provider lane. See [raw binary analysis](raw-binary.md).
cleanup. Six-byte x86 and x86_64 controls also verify supported compiler
specifications, independent mappings, unchanged bytes, and decompilation of a
known return value. AArch64 remains covered by deterministic tests rather than
this real-provider lane. See [raw binary analysis](raw-binary.md).
## Apple Interface Builder archives
@@ -1379,7 +1386,8 @@ generation. It installs nothing. Without configuration, the default invocation
reports a named skip; `--require-tools` makes missing configuration fail.
The verifier creates and validates real modules with custom sections, multiple
import/export kinds, escaped names and empty modules. It compares exact WAT,
import/export kinds, escaped names, an empty module, and a bulk-memory
DataCount section. It compares exact WAT,
section output and selected/tool digests through the compiled CLI and real MCP
SDK, validates advertised schemas, checks identical CLI/MCP Evidence IDs and
session bundle readback, retains invalid/truncated module diagnostics, preserves
+104 -88
View File
@@ -1,4 +1,12 @@
#!/bin/bash
{
# Truncation guard: this brace group opens on line 2 and closes on the last
# line, so bash must read the entire file before it runs any command. On the
# `curl ... | bash` path, a download cut off anywhere before the final `}` is
# an unterminated compound command, which bash reports as a syntax error at
# EOF without running anything. Without the guard, a cut right after `set`
# would run the bare builtin and print every shell variable, and a cut right
# after `main` would run the installer without the caller's flags.
set -euo pipefail
PACKAGE="rea-agents"
@@ -21,104 +29,112 @@ usage() {
cat <<'EOF'
Usage: install.sh [--version VERSION] [--dry-run] [--no-setup] [--no-prompt] [--verbose]
Installs only the REA CLI. Run `rea setup` separately to configure Hopper and agents.
Installs the REA CLI, then starts `rea setup` interactively to configure Hopper
and agents when a terminal is available. Pass --no-setup (or --no-prompt) to
skip setup and run `rea setup` later.
EOF
}
while (($# > 0)); do
case "$1" in
--version)
(($# >= 2)) || fail "--version requires a value."
version="$2"
shift 2
;;
--dry-run)
dry_run=true
shift
;;
--no-setup|--no-prompt)
start_setup=false
shift
;;
--verbose)
verbose=true
shift
;;
--help|-h)
usage
exit 0
;;
*) fail "unknown option: $1" ;;
esac
done
main() {
while (($# > 0)); do
case "$1" in
--version)
(($# >= 2)) || fail "--version requires a value."
version="$2"
shift 2
;;
--dry-run)
dry_run=true
shift
;;
--no-setup|--no-prompt)
start_setup=false
shift
;;
--verbose)
verbose=true
shift
;;
--help|-h)
usage
exit 0
;;
*) fail "unknown option: $1" ;;
esac
done
if [[ "$verbose" == true ]]; then
set -x
fi
if [[ "$verbose" == true ]]; then
set -x
fi
platform="$(uname -s)"
[[ "$platform" == "Darwin" || "$platform" == "Linux" ]] || fail "REA supports macOS and selected 64-bit Linux distributions."
command -v curl >/dev/null 2>&1 || fail "curl is required. Install curl, then rerun this installer."
command -v node >/dev/null 2>&1 || fail "Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+ is required; install it from https://nodejs.org and retry."
command -v npm >/dev/null 2>&1 || fail "npm is required; install it with Node.js and retry."
platform="$(uname -s)"
[[ "$platform" == "Darwin" || "$platform" == "Linux" ]] || fail "REA supports macOS and selected 64-bit Linux distributions."
command -v curl >/dev/null 2>&1 || fail "curl is required. Install curl, then rerun this installer."
command -v node >/dev/null 2>&1 || fail "Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+ is required; install it from https://nodejs.org and retry."
command -v npm >/dev/null 2>&1 || fail "npm is required; install it with Node.js and retry."
node_version="$(node -p 'process.versions.node' 2>/dev/null || true)"
IFS=. read -r node_major node_minor _node_patch <<<"$node_version"
[[ "$node_major" =~ ^[0-9]+$ && "$node_minor" =~ ^[0-9]+$ ]] || fail "the active Node.js version could not be read. Check that node works and is on PATH, then retry."
if ! [[ "$_node_patch" =~ ^[0-9]+(\+[0-9A-Za-z.-]+)?$ ]] ||
! ((node_major == 22 && node_minor >= 19 || node_major == 24 && node_minor >= 11 || node_major >= 26)); then
fail "Node.js $node_version is unsupported; use Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+."
fi
node_version="$(node -p 'process.versions.node' 2>/dev/null || true)"
IFS=. read -r node_major node_minor _node_patch <<<"$node_version"
[[ "$node_major" =~ ^[0-9]+$ && "$node_minor" =~ ^[0-9]+$ ]] || fail "the active Node.js version could not be read. Check that node works and is on PATH, then retry."
if ! [[ "$_node_patch" =~ ^[0-9]+(\+[0-9A-Za-z.-]+)?$ ]] ||
! ((node_major == 22 && node_minor >= 19 || node_major == 24 && node_minor >= 11 || node_major >= 26)); then
fail "Node.js $node_version is unsupported; use Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+."
fi
if [[ -n "$version" ]]; then
[[ "$version" =~ ^${version_pattern}$ ]] || fail "version must be an exact semantic version."
else
release_json="$(curl -fsSL --proto '=https' --tlsv1.2 --retry 3 -H 'Accept: application/vnd.github+json' "https://api.github.com/repos/$REPOSITORY/releases/latest")" || fail "the latest REA release could not be resolved. Check network access or pass --version VERSION, then retry."
tag="$(printf '%s' "$release_json" | node -e 'let s="";process.stdin.on("data",c=>s+=c).on("end",()=>{const v=JSON.parse(s).tag_name;if(typeof v!=="string")process.exit(1);process.stdout.write(v)})' 2>/dev/null)" || fail "the release response was invalid. Retry later or pass --version VERSION."
[[ "$tag" =~ ^rea-agents-(${version_pattern})$ ]] || fail "the latest release tag was invalid. Retry later or pass --version VERSION."
version="${BASH_REMATCH[1]}"
fi
if [[ -n "$version" ]]; then
[[ "$version" =~ ^${version_pattern}$ ]] || fail "version must be an exact semantic version."
else
release_json="$(curl -fsSL --proto '=https' --tlsv1.2 --retry 3 -H 'Accept: application/vnd.github+json' "https://api.github.com/repos/$REPOSITORY/releases/latest")" || fail "the latest REA release could not be resolved. Check network access or pass --version VERSION, then retry."
tag="$(printf '%s' "$release_json" | node -e 'let s="";process.stdin.on("data",c=>s+=c).on("end",()=>{const v=JSON.parse(s).tag_name;if(typeof v!=="string")process.exit(1);process.stdout.write(v)})' 2>/dev/null)" || fail "the release response was invalid. Retry later or pass --version VERSION."
[[ "$tag" =~ ^rea-agents-(${version_pattern})$ ]] || fail "the latest release tag was invalid. Retry later or pass --version VERSION."
version="${BASH_REMATCH[1]}"
fi
# npm's exact-version resolver cleans build metadata from the requested version,
# so it may install the base release and make the post-install identity check fail.
if [[ "$version" =~ \+ ]]; then
fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry."
fi
# npm's exact-version resolver cleans build metadata from the requested version,
# so it may install the base release and make the post-install identity check fail.
if [[ "$version" =~ \+ ]]; then
fail "npm cannot install an exact version with build metadata. Pass a version without build metadata, then retry."
fi
install_args=(install --global)
if [[ "$platform" == "Linux" ]]; then
install_args+=(--prefix "$HOME/.local")
install_bin="$HOME/.local/bin/rea"
else
npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry."
install_bin="$npm_prefix/bin/rea"
fi
install_args+=("$PACKAGE@$version")
install_args=(install --global)
if [[ "$platform" == "Linux" ]]; then
install_args+=(--prefix "$HOME/.local")
install_bin="$HOME/.local/bin/rea"
else
npm_prefix="$(npm prefix --global)" || fail "the npm global prefix could not be read. Repair the npm configuration, then retry."
install_bin="$npm_prefix/bin/rea"
fi
install_args+=("$PACKAGE@$version")
printf 'REA install plan\n'
printf ' Version: %s\n' "$version"
printf ' Runtime: Node.js %s\n' "$node_version"
printf ' Command:'
printf ' %q' npm "${install_args[@]}"
printf '\n'
printf ' Binary: %s\n' "$install_bin"
printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')"
printf 'REA install plan\n'
printf ' Version: %s\n' "$version"
printf ' Runtime: Node.js %s\n' "$node_version"
printf ' Command:'
printf ' %q' npm "${install_args[@]}"
printf '\n'
printf ' Binary: %s\n' "$install_bin"
printf ' Setup: %s\n' "$([[ "$start_setup" == true ]] && printf 'start when a terminal is available' || printf 'skipped')"
if [[ "$dry_run" == true ]]; then
printf 'Dry run complete; no changes made.\n'
exit 0
fi
if [[ "$dry_run" == true ]]; then
printf 'Dry run complete; no changes made.\n'
exit 0
fi
printf 'Installing %s@%s...\n' "$PACKAGE" "$version"
npm "${install_args[@]}" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
[[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry."
installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry."
[[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry."
printf 'Installing %s@%s...\n' "$PACKAGE" "$version"
npm "${install_args[@]}" || fail "npm could not install REA. Check registry access and npm permissions, then retry."
[[ -x "$install_bin" ]] || fail "npm completed without installing the rea command. Check the npm global bin directory and PATH, then retry."
installed_version="$("$install_bin" --version 2>/dev/null | tr -d '[:space:]')" || fail "the installed REA version could not be read. Reinstall the requested version, then retry."
[[ "$installed_version" == "$version" ]] || fail "installed version $installed_version does not match $version. Reinstall the requested version, then retry."
printf 'REA %s is installed.\n' "$version"
if [[ "$start_setup" == true && -r /dev/tty && -w /dev/tty && -t 1 ]]; then
printf 'Starting REA setup...\n'
"$install_bin" setup </dev/tty >/dev/tty
else
printf 'Run %s setup to configure Hopper and agents.\n' "$install_bin"
fi
printf 'REA %s is installed.\n' "$version"
if [[ "$start_setup" == true && -r /dev/tty && -w /dev/tty && -t 1 ]]; then
printf 'Starting REA setup...\n'
"$install_bin" setup </dev/tty >/dev/tty
else
printf 'Run %s setup to configure Hopper and agents.\n' "$install_bin"
fi
}
main "$@"
exit $?
}
+4
View File
@@ -51,10 +51,13 @@
"bridge/process/ProcessRunTokenReader.swift",
"bridge/process/ProcessRunTokenReaderMain.swift",
"bridge/ghidra/ReaGhidraBridge.java",
"bridge/ghidra/ReaGhidraBundleActivator.java",
"bridge/ghidra/ReaGhidraExtensions.java",
"bridge/ghidra/ReaGhidraPrepareCom.java",
"bridge/ghidra/ReaGhidraApplySeeds.java",
"bridge/ghidra/ReaGhidraAnalysisOptions.java",
"bridge/ghidra/ReaGhidraNoReturnFix.java",
"bridge/ghidra/ReaGhidraMipsSoftFloat.java",
"bridge/ghidra/ReaGhidraPrepareRaw.java",
"bridge/android/ReaJadxBridge.java",
"bridge/mitmproxy/capture.py",
@@ -162,6 +165,7 @@
"verify:ghidra": "npm run build:cached && node scripts/verify-real-ghidra.mjs",
"verify:ghidra:seeds": "npm run build:cached && node scripts/verify/ghidra/seeds.mjs",
"verify:ghidra:mips": "npm run build:cached && node scripts/verify-real-ghidra-mips.mjs",
"verify:ghidra:mips-relocatable": "npm run build:cached && node scripts/verify-real-ghidra-mips-relocatable.mjs",
"verify:ghidra:swift": "npm run build:cached && node scripts/verify-real-ghidra-swift.mjs",
"verify:ida": "npm run build:cached && node scripts/verify-real-ida.mjs",
"verify:ghidra:switch": "npm run build:cached && node scripts/verify-real-ghidra-switch.mjs",
+16
View File
@@ -401,6 +401,22 @@ class WebsiteTranslationChecks(unittest.TestCase):
server.shutdown()
worker.join()
def test_every_reading_page_offers_light_and_dark_theme_colors(self):
site = ROOT / "website/_site"
english_pages = reading_pages(ROOT / "website/public")
self.assertTrue(english_pages)
for language in languages(ROOT):
code = language["code"]
for original in english_pages:
relative = original.relative_to(ROOT / "website/public")
for path in {original, site / relative if code == "en" else site / code / relative}:
with self.subTest(language=code, page=str(path.relative_to(ROOT))):
doc = Document(path.read_text())
media = sorted(node.attrs.get("media") for node in doc.nodes if node.tag == "meta" and node.attrs.get("name") == "theme-color")
self.assertEqual(media, ["(prefers-color-scheme: dark)", "(prefers-color-scheme: light)"])
styles = (ROOT / "website/public/assets/styles.css").read_text()
self.assertTrue("@media (prefers-color-scheme: dark)" in styles, "styles.css has no dark color scheme block")
if __name__ == "__main__":
unittest.main()
+6 -5
View File
@@ -13,10 +13,6 @@ import { exec } from "./lib/verify-package-core.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { createAnthropicModelFixture } from "./verify/clients/anthropic-model-fixture.mjs";
assert(
process.platform !== "win32",
"Native Claude Code verifier requires POSIX; Windows is unverified",
);
const repo = fileURLToPath(new URL("..", import.meta.url));
const runtimeRoot = process.env.REA_VERIFY_RUNTIME_ROOT ?? repo;
const command = process.env.REA_VERIFY_CLAUDE_COMMAND ?? "claude";
@@ -46,8 +42,13 @@ try {
await exec(command, ["--version"], { env: environment, timeout: 30_000 })
).stdout.trim();
} catch (cause) {
// npm installs put only a claude.cmd shim on PATH, which Node does not spawn without a shell.
const windowsHint =
process.platform === "win32"
? " On Windows, select claude.exe; npm installs provide it under node_modules/@anthropic-ai/claude-code/bin."
: "";
throw new Error(
`Claude Code preflight failed for ${command}; install Claude Code or set REA_VERIFY_CLAUDE_COMMAND. ${String(cause)}`,
`Claude Code preflight failed for ${command}; install Claude Code or set REA_VERIFY_CLAUDE_COMMAND.${windowsHint} ${String(cause)}`,
);
}
const target = join(lab, "fixture α测试");
+22 -3
View File
@@ -14,7 +14,7 @@ import {
writeFile,
} from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path";
import { basename, dirname, join, resolve } from "node:path";
import { promisify } from "node:util";
import { fileURLToPath, pathToFileURL } from "node:url";
@@ -172,11 +172,13 @@ const digest = (bytes) => createHash("sha256").update(bytes).digest("hex");
const runtimeParent = join(workspace, "runtime with spaces");
await mkdir(runtimeParent);
const callerDirectory = join(workspace, "caller cwd with spaces");
// Keep sorted for comparison with the final caller-directory listing.
const scriptCollisions = [
"ReaGhidraBridge.java",
"ReaGhidraPrepareCom.java",
"ReaGhidraAnalysisOptions.java",
"ReaGhidraApplySeeds.java",
"ReaGhidraBridge.java",
"ReaGhidraNoReturnFix.java",
"ReaGhidraPrepareCom.java",
"ReaGhidraPrepareRaw.java",
];
await mkdir(callerDirectory);
@@ -291,6 +293,23 @@ try {
await client.connect(transport);
await call("open_binary", { path: target });
await call("binary_overview", {});
const activators = (
await readdir(runtimeParent, { recursive: true })
).filter((path) => basename(path) === "ReaGhidraBundleActivator.class");
assert.equal(activators.length, 1);
const [activator] = activators;
assert.ok(activator !== undefined);
const bundleDirectory = dirname(join(runtimeParent, activator));
const manifest = await readFile(
join(bundleDirectory, "META-INF", "MANIFEST.MF"),
"utf8",
);
assert.match(manifest, /^Bundle-Activator: ReaGhidraBundleActivator\r?$/mu);
await assert.rejects(
readFile(join(bundleDirectory, "GeneratedActivator.class")),
{ code: "ENOENT" },
);
report.bundleActivator = "ReaGhidraBundleActivator";
assert.equal((await call("list_documents", {})).length, 1);
await call("list_names", {});
const procedures = await call("list_procedures", {});
+7 -1
View File
@@ -5,10 +5,16 @@ import { promisify } from "node:util";
const exec = promisify(execFile);
const packedHopperBridge = "package/bridge/hopper_bridge.py";
const packedGhidraBridge = "package/bridge/ghidra/ReaGhidraBridge.java";
const packedGhidraActivator =
"package/bridge/ghidra/ReaGhidraBundleActivator.java";
/** Check shipped bridge files; real-provider lanes verify runtime behavior. */
export async function verifyPackedBridge({ workspace, tarball, packedFiles }) {
for (const path of [packedHopperBridge, packedGhidraBridge]) {
for (const path of [
packedHopperBridge,
packedGhidraBridge,
packedGhidraActivator,
]) {
if (!packedFiles.includes(path)) throw new Error(`package omitted ${path}`);
}
await exec("tar", ["-xf", tarball, "-C", workspace]);
+333
View File
@@ -0,0 +1,333 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdtemp, mkdir, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { Client } from "@modelcontextprotocol/client";
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
import { parseBinaryTarget } from "../dist/application/BinaryTargetResolver.js";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
import { functionDossierSchema } from "../dist/domain/hopperValues.js";
import { inspectGhidraInstallation } from "../dist/ghidra/GhidraInstallation.js";
import { buildMipsRelocatableFixture } from "../tests/conformance/ghidra/mips-fixture.mjs";
import {
assertMipsRelocations,
inspectMipsRelocatableReadelf,
} from "../tests/conformance/ghidra/mips-relocatable-checks.mjs";
import { requireMcpOperationResult } from "./lib/mcp-verifier-results.mjs";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
// Optional cross-target lane for ET_REL objects. No tool acquisition, no
// linking and no target execution.
if (process.argv.length > 2)
throw new Error(
"Usage: node scripts/verify-real-ghidra-mips-relocatable.mjs",
);
if (!["linux", "darwin"].includes(process.platform))
throw new Error(
"MIPS verification requires a supported Linux/macOS Ghidra host",
);
if (process.env.GHIDRA_INSTALL_DIR === undefined)
throw new Error("Missing prerequisite: caller-supplied GHIDRA_INSTALL_DIR");
const installation = inspectGhidraInstallation({
environment: process.env,
installDir: process.env.GHIDRA_INSTALL_DIR,
...(process.env.JAVA_HOME === undefined
? {}
: { javaHome: process.env.JAVA_HOME }),
});
if (installation.status !== "available")
throw new Error(
`Ghidra prerequisite unavailable: ${JSON.stringify(installation)}`,
);
const execute = promisify(execFile);
const entrypoint = fileURLToPath(new URL("./rea.mjs", import.meta.url));
const run = createVerifierRun();
const workspace = await mkdtemp(join(tmpdir(), "rea-mips-rel-proof-"));
const runtime = join(workspace, "runtime");
await mkdir(runtime);
const env = {
PATH: process.env.PATH ?? "/usr/bin:/bin",
TMPDIR: runtime,
REA_LOG_LEVEL: "silent",
REA_ANALYSIS_PROVIDER: "ghidra",
REA_PROCESS_RUN_ID: run.run_id,
GHIDRA_INSTALL_DIR: installation.installDir,
JAVA_HOME: installation.javaHome,
HOPPER_LAUNCHER_PATH: "/rea-unconfigured-deep-provider/hopper",
};
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
/** Load every relocated word through the public read_bytes operation. */
async function loadedWords(call, independent, segments, byteOrder) {
const words = new Map();
for (const relocation of independent.relocations) {
const section = [...independent.sections.values()].find(
(item) => item.name === relocation.section,
);
if (!section?.allocated) continue;
const segment = segments.find((item) => item.name === section.name);
assert.ok(segment, `Ghidra segment ${section.name}`);
const address = BigInt(segment.start) + BigInt(relocation.offset);
const read = await call("read_bytes", {
address: `0x${address.toString(16)}`,
length: 4,
});
assert.equal(read.returned_bytes, 4);
const bytes = Buffer.from(read.bytes_hex, "hex");
words.set(
address,
byteOrder === "little" ? bytes.readUInt32LE(0) : bytes.readUInt32BE(0),
);
}
return words;
}
async function verifyFixture(byteOrder, kernelModule) {
const fixture = await buildMipsRelocatableFixture(
workspace,
byteOrder,
kernelModule,
);
const independent = await inspectMipsRelocatableReadelf(
fixture.path,
byteOrder,
fixture.float_abi,
);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
assert.deepEqual(
resolved.value.mips,
independent.mips,
"REA ELF/ABI interpretation disagrees with GNU readelf",
);
assert.equal(resolved.value.sha256, fixture.sha256);
const label = `${byteOrder}${kernelModule ? "-module" : ""}`;
const snapshotPath = join(workspace, `${label}.snapshot.json`);
const transport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
env,
stderr: "pipe",
});
const client = new Client({ name: "rea-real-mips-rel-proof", version: "1" });
let stderr = "";
transport.stderr?.on("data", (chunk) => {
stderr = (stderr + chunk.toString()).slice(-65536);
});
const call = async (name, args = {}) =>
requireMcpOperationResult(
await client.callTool({ name, arguments: args }, undefined, {
timeout: 180000,
}),
name,
);
try {
await client.connect(transport);
const target = await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
});
assert.equal(target.architecture, "mips");
assert.equal(target.sha256, fixture.sha256);
const image = (await call("inspect_native_load_image")).observations;
assert.equal(
image.language_id,
`MIPS:${byteOrder === "little" ? "LE" : "BE"}:32:default`,
);
assert.equal(image.source_files.length, 1);
assert.equal(image.source_files[0].original_sha256, fixture.sha256);
assert.notEqual(
image.source_files[0].modified_sha256,
fixture.sha256,
"Relocations must change the loaded bytes",
);
const names = await call("list_names");
const segments = await call("list_segments");
const coverage = assertMipsRelocations({
facts: independent,
fileBytes: await readFile(fixture.path),
segments,
mappings: image.mappings,
ghidraRelocations: image.relocations,
words: await loadedWords(call, independent, segments, byteOrder),
names,
byteOrder,
});
assert.equal(coverage.unplaced, 0, "Fixture references an empty section");
assert.ok(coverage.shared_hi16, "No HI16 shared by several LO16s");
assert.ok(coverage.lo16_carry, "No LO16 >= 0x8000 forced a HI16 carry");
assert.ok(coverage.external_code, "No external code relocation");
assert.ok(coverage.external_data, "No external data relocation");
assert.ok(coverage.types.R_MIPS_32 > 0, "No R_MIPS_32 relocation");
assert.equal(
coverage.types.R_MIPS_26 === undefined,
kernelModule,
"Long calls must replace every R_MIPS_26 with HI16/LO16",
);
const procedures = await call("list_procedures");
const named = (name) => {
const found = procedures.filter((item) => item.value === name);
assert.equal(found.length, 1, `Procedure ${name}`);
return found[0].address;
};
const entry = named("rea_mips_rel_entry");
const callees = await call("procedure_callees", { procedure: entry });
for (const callee of [
named("rea_mips_rel_leaf"),
named("rea_mips_rel_far"),
])
assert.ok(callees.includes(callee), `Missing direct call to ${callee}`);
// Ghidra lists the external function and its EXTERNAL-block thunk.
assert.ok(
procedures.some(
(item) =>
item.value === "<EXTERNAL>::rea_mips_rel_external" &&
callees.includes(item.address),
),
"Missing call to the external function",
);
const dossier = functionDossierSchema.parse(
await call("analyze_function", { procedure: entry }),
);
for (const name of [
"rea_mips_rel_leaf",
"rea_mips_rel_far",
"rea_mips_rel_external",
])
assert.ok(dossier.pseudocode.includes(name), `Pseudocode lost ${name}`);
// powf's float prototype comes from Ghidra's C library archive; soft-float
// builds pass 1.5f/2.5f in $a0/$a1, hard-float builds in $f12/$f14.
const scaled = functionDossierSchema.parse(
await call("analyze_function", {
procedure: named("rea_mips_rel_scaled"),
}),
);
assert.match(scaled.pseudocode, /\bpowf\(1\.5,2\.5\)/u);
assert.doesNotMatch(scaled.pseudocode, /\bin_f\d+\b/u);
await call("close_binary", { snapshot_path: snapshotPath });
const snapshot = parseAnalysisSnapshot(
JSON.parse(await readFile(snapshotPath, "utf8")),
);
const parameters = snapshot.binding.analysis_profile.parameters;
assert.equal(parameters.mips_elf.type, 1);
assert.equal(
parameters.mips_support_lane,
"elf32-rel-o32-arch32r2-standard-v1",
);
assert.equal(
parameters.mips_soft_float_storage,
kernelModule ? "o32-soft-float-prototype-v1" : undefined,
);
assert.ok(
snapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
item.includes("Relocatable MIPS ELF"),
),
),
"Relocatable limitation missing from retained Evidence",
);
assert.equal(
snapshot.evidence_bundle.records.some((record) =>
parseEvidence(record).limitations.some((item) =>
/Soft-float prototype outcome: [1-9]\d* functions/u.test(item),
),
),
kernelModule,
"Soft-float prototype must cover the archive-typed external in module builds",
);
await call("open_binary", {
path: fixture.path,
provider_id: "ghidra",
snapshot_path: snapshotPath,
});
assert.deepEqual(await call("list_procedures"), procedures);
await call("close_binary");
// Same public operation through CLI, not a private injected provider.
const cli = await execute(
process.execPath,
[
entrypoint,
"function",
fixture.path,
"rea_mips_rel_entry",
"--provider",
"ghidra",
"--json",
],
{ env, timeout: 240000, maxBuffer: 16 * 1024 * 1024 },
);
const evidence = parseEvidence(JSON.parse(cli.stdout));
assert.equal(evidence.subject?.architecture, "mips");
assert.equal(evidence.subject?.digest.sha256, fixture.sha256);
assert.equal(
functionDossierSchema.parse(evidence.normalized_result).procedure.address,
dossier.procedure.address,
);
assert.equal(
createHash("sha256")
.update(await readFile(fixture.path))
.digest("hex"),
fixture.sha256,
"Analysis modified the source object",
);
return {
...fixture,
independent_reader: {
command: independent.command,
version: independent.version,
},
effective_language: image.language_id,
relocation_coverage: coverage,
procedures: procedures.length,
status: "passed",
};
} catch (cause) {
throw new Error(
`MIPS ${label} relocatable verification failed; MCP stderr: ${stderr}`,
{ cause },
);
} finally {
try {
await client.close();
} finally {
await transport.close();
}
}
}
const reports = [];
let primaryFailure;
try {
for (const [byteOrder, kernelModule] of [
["little", false],
["big", false],
["little", true],
])
reports.push(await verifyFixture(byteOrder, kernelModule));
} catch (cause) {
primaryFailure = cause;
}
const completed = await completeVerifierRun(run);
if (
completed.process_lineage.status !== "verified" ||
completed.process_lineage.descendants.length !== 0
)
throw new Error(
`Cleanup unverified; retained diagnostic workspace: ${workspace}`,
{ cause: primaryFailure },
);
if (primaryFailure !== undefined)
throw new Error(
`MIPS relocatable verification failed; retained workspace: ${workspace}`,
{ cause: primaryFailure },
);
await rm(workspace, { recursive: true, force: true });
console.log(JSON.stringify({ ...completed, fixtures: reports }, null, 2));
+83 -7
View File
@@ -15,6 +15,7 @@ import {
assertMipsLoadedImage,
assertMipsProbe,
assertMipsGlobal,
assertMipsSoftFloatStorage,
} from "../tests/conformance/ghidra/mips-checks.mjs";
import { parseEvidence } from "../dist/domain/evidence.js";
import { parseAnalysisSnapshot } from "../dist/domain/analysisSnapshot.js";
@@ -77,16 +78,64 @@ const env = {
for (const setting of ["GHIDRA_HEADLESS_MAXMEM", "GHIDRA_MAXMEM"]) {
if (process.env[setting] !== undefined) env[setting] = process.env[setting];
}
/**
* Typed float signatures must use soft-float o32 storage, and callers must see
* the source constants through it rather than FPU inputs.
*/
async function verifySoftFloat(call, symbols, byteOrder) {
const boundaries = {};
for (const name of [
"rea_mips_soft_pick",
"rea_mips_soft_wide",
"rea_mips_soft_mixed",
"rea_mips_soft_struct",
]) {
const api = await call("inspect_native_api", { procedure: symbols[name] });
boundaries[name] = api.boundary;
}
assertMipsSoftFloatStorage(boundaries, byteOrder);
const caller = functionDossierSchema.parse(
await call("analyze_function", {
procedure: symbols.rea_mips_soft_caller,
}),
);
// Includes the struct call and the indirect call through a typed pointer.
assert.doesNotMatch(caller.pseudocode, /\bin_f\d+\b/u);
for (const literal of [
"1.5",
"2.5",
"3.25",
"4.75",
"5.5",
"6.125",
"7.5",
"8.5",
"9.5",
])
assert.ok(
caller.pseudocode.includes(literal),
`Soft-float caller lost argument ${literal}`,
);
return { storage: boundaries };
}
const reports = [];
let primaryFailure;
try {
for (const byteOrder of psp ? ["little"] : ["little", "big"]) {
for (const [byteOrder, floatAbi] of psp
? [["little", "hard"]]
: [
["little", "hard"],
["big", "hard"],
["little", "soft"],
["big", "soft"],
]) {
const fixture = psp
? await buildPspFixture(workspace)
: await buildMipsFixture(workspace, byteOrder);
: await buildMipsFixture(workspace, byteOrder, floatAbi);
const independent = psp
? fixture
: await inspectMipsReadelf(fixture.path, byteOrder);
: await inspectMipsReadelf(fixture.path, byteOrder, floatAbi);
const resolved = await parseBinaryTarget(fixture.path);
if (!resolved.ok) throw resolved.error;
if (psp) {
@@ -104,7 +153,10 @@ try {
);
}
assert.equal(resolved.value.sha256, fixture.sha256);
const snapshotPath = join(workspace, `${byteOrder}.snapshot.json`);
const snapshotPath = join(
workspace,
`${byteOrder}-${floatAbi}.snapshot.json`,
);
const transport = new StdioClientTransport({
command: process.execPath,
args: [entrypoint, "mcp"],
@@ -238,6 +290,10 @@ try {
callees.includes(leaf.address),
"Expected direct call not recovered",
);
const softFloat =
floatAbi === "soft"
? await verifySoftFloat(call, independent.symbols, byteOrder)
: undefined;
const assembly = await call("procedure_assembly", {
procedure: entry.address,
});
@@ -293,6 +349,19 @@ try {
},
"Snapshot profile did not retain the independently inspected ELF/ABI declaration",
);
assert.equal(
snapshot.binding.analysis_profile.parameters.mips_soft_float_storage,
floatAbi === "soft" ? "o32-soft-float-prototype-v1" : undefined,
);
assert.equal(
snapshot.evidence_bundle.records.some((record) =>
/Soft-float prototype outcome: 4 functions and [1-9]\d* function-pointer definitions use __rea_soft_float\./u.test(
JSON.stringify(record),
),
),
floatAbi === "soft",
"Soft-float prototype outcome missing from, or unexpected in, retained Evidence",
);
assert.ok(snapshot.evidence_bundle.records.length > 0);
if (psp) {
assert.equal(
@@ -419,7 +488,10 @@ try {
assert.ok(read32(offset + 4) + read32(offset + 16) <= cut);
}
const carvedBytes = sourceBytes.subarray(0, cut);
const carvedPath = join(workspace, `${byteOrder}-carved.elf`);
const carvedPath = join(
workspace,
`${byteOrder}-${floatAbi}-carved.elf`,
);
const carvedSha256 = createHash("sha256")
.update(carvedBytes)
.digest("hex");
@@ -460,7 +532,7 @@ try {
assert.ok(carvedDossier.pseudocode.trim().length > 0);
const carvedSnapshotPath = join(
workspace,
`${byteOrder}-carved.snapshot.json`,
`${byteOrder}-${floatAbi}-carved.snapshot.json`,
);
await call("close_binary", { snapshot_path: carvedSnapshotPath });
const carvedSnapshot = parseAnalysisSnapshot(
@@ -523,7 +595,11 @@ try {
: "conditional branch destination",
"global word and marker bytes",
"CLI/MCP and snapshot identity",
...(floatAbi === "soft"
? ["soft-float o32 parameter/return storage and caller constants"]
: []),
],
...(softFloat === undefined ? {} : { soft_float: softFloat }),
procedures: procedures.length,
carved: carved ?? {
status: "not_run",
@@ -534,7 +610,7 @@ try {
});
} catch (cause) {
throw new Error(
`MIPS ${byteOrder} verification failed; MCP stderr: ${stderr}`,
`MIPS ${byteOrder} ${floatAbi}-float verification failed; MCP stderr: ${stderr}`,
{ cause },
);
} finally {
+54 -2
View File
@@ -53,6 +53,15 @@ const env = {
...(process.env.JAVA_HOME === undefined
? {}
: { JAVA_HOME: process.env.JAVA_HOME }),
...(process.env.GHIDRA_HEADLESS_MAXMEM === undefined
? {}
: { GHIDRA_HEADLESS_MAXMEM: process.env.GHIDRA_HEADLESS_MAXMEM }),
...(process.env.REA_GHIDRA_DISABLED_ANALYZERS === undefined
? {}
: {
REA_GHIDRA_DISABLED_ANALYZERS:
process.env.REA_GHIDRA_DISABLED_ANALYZERS,
}),
};
const transport = new StdioClientTransport({
command: process.execPath,
@@ -160,6 +169,48 @@ try {
assert.equal(cli.normalized_result.base_address, "0x8000000");
assert.equal(cli.analysis_profile.digest, profiles[0]);
assert.deepEqual(await readFile(path), bytes);
const x86Targets = [];
for (const architecture of ["x86", "x86_64"]) {
const x86Bytes = Buffer.from("b82a000000c3", "hex"); // mov eax, 42; ret.
const x86Sha256 = createHash("sha256").update(x86Bytes).digest("hex");
const x86Path = join(workspace, `${architecture}.bin`);
await writeFile(x86Path, x86Bytes);
await call("open_binary", {
path: x86Path,
format: "raw-binary",
architecture,
entry_point: "0x1000",
});
const image = await call("inspect_native_load_image", {}, x86Sha256);
assert.equal(
image.normalized_result.status,
"verified",
JSON.stringify(image),
);
assert.equal(image.normalized_result.base_address, "0x1000");
assert.equal(image.normalized_result.observations.compiler_spec_id, "gcc");
assert.equal(image.analysis_profile.parameters.compiler_spec_id, "gcc");
const memory = await call(
"read_bytes",
{ address: "0x1000", length: x86Bytes.length },
x86Sha256,
);
assert.equal(memory.normalized_result.bytes_hex, x86Bytes.toString("hex"));
const pseudo = await call(
"procedure_pseudo_code",
{ procedure: "0x1000" },
x86Sha256,
);
assert.match(pseudo.normalized_result, /\b(?:42|0x2a)\b/iu);
await call("close_binary");
assert.deepEqual(await readFile(x86Path), x86Bytes);
x86Targets.push({
architecture,
sha256: x86Sha256,
compiler_spec_id: "gcc",
return_42: true,
});
}
report = {
ok: true,
mocked: false,
@@ -179,6 +230,7 @@ try {
invalid_entry_preserves_active_session: true,
load_images_verified: true,
original_unchanged: true,
x86_targets: x86Targets,
};
} catch (error) {
if (stderr.trim()) process.stderr.write(stderr);
@@ -199,7 +251,7 @@ process.stdout.write(
}) + "\n",
);
async function call(name, args = {}) {
async function call(name, args = {}, targetSha256 = sha256) {
const result = await client.callTool(
{ name, arguments: args },
{ timeout: 240000 },
@@ -213,7 +265,7 @@ async function call(name, args = {}) {
)
return projected;
const evidence = parseEvidence(result.structuredContent);
assert.equal(evidence.subject.digest.sha256, sha256);
assert.equal(evidence.subject.digest.sha256, targetSha256);
assert.equal(evidence.subject.format, "raw-binary");
return evidence;
}
+25
View File
@@ -122,6 +122,15 @@ try {
new URL("../tests/fixtures/wasm/custom-names.wat", import.meta.url),
"utf8",
),
`(module
(memory 1)
(data "x")
(func (export "init")
i32.const 0
i32.const 0
i32.const 1
memory.init 0
data.drop 0))\n`,
];
for (const [index, source] of sources.entries()) {
const fixtureRoot = join(root.path, `fixture-${index}`);
@@ -220,6 +229,22 @@ try {
'"unicode-é\\\t"',
],
);
if (index === 4) {
assert.match(
headers,
/^DataCount start=0x[\da-f]+ end=0x[\da-f]+ \(size=0x[\da-f]+\) count: 1$/m,
);
assert.deepEqual(
report.sections.find(({ kind }) => kind === "DataCount"),
{
kind: "DataCount",
start: 0x23,
end: 0x24,
bytes: 1,
description: "count: 1",
},
);
}
const bundleReply = await client.callTool({
name: "get_evidence_bundle",
arguments: {},
@@ -0,0 +1,147 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { z } from "zod";
import { keyedArchiveResultSchema } from "../../../dist/domain/apple/keyedArchive.js";
const rowSchema = z.object({
container: z.enum(["roots", "objects"]),
root: z.string(),
objectId: z.number().int().nonnegative().optional(),
path: z.array(z.union([z.string(), z.number().int().nonnegative()])),
kind: z.enum(["integer", "real"]),
literal: z.string(),
});
const archiveSchema = z.object({
filename: z.string(),
sha256: z.string().regex(/^[a-f0-9]{64}$/u),
rows: z.array(rowSchema),
});
const oracleSchema = z.object({ xml: archiveSchema, binary: archiveSchema });
const formats = ["xml", "binary"];
const requiredRows = [
{ root: "unsafeSigned", kind: "integer", literal: "9007199254740993" },
{
root: "unsafeNegative",
kind: "integer",
literal: "-9223372036854775808",
},
{ root: "safeInteger", kind: "integer", literal: "9007199254740991" },
{ root: "safeNegative", kind: "integer", literal: "-42" },
{ root: "integralReal", kind: "real", literal: "9007199254740992" },
{
root: "boxedUnsigned",
kind: "integer",
literal: "9007199254740993",
container: "objects",
},
{
root: "boxedReal",
kind: "real",
literal: "9007199254740992",
container: "objects",
},
];
const valueAt = (value, path) => {
for (const key of path) {
if (value === null || typeof value !== "object") return undefined;
value = value[key];
}
return value;
};
const isUnsafeInteger = (row) =>
row.kind === "integer" &&
(BigInt(row.literal) > BigInt(Number.MAX_SAFE_INTEGER) ||
BigInt(row.literal) < BigInt(Number.MIN_SAFE_INTEGER));
/** Verify the actual Foundation output and the production CLI/MCP result. */
export async function verifyKeyedArchiveIntegerArchives(root) {
const oracle = oracleSchema.parse(
JSON.parse(await readFile(join(root, "oracle.json"), "utf8")),
);
const archives = [];
for (const format of formats) {
const archive = oracle[format];
for (const required of requiredRows) {
assert.ok(
archive.rows.some(
(row) =>
row.root === required.root &&
row.kind === required.kind &&
row.literal === required.literal &&
(required.container === undefined ||
row.container === required.container),
),
`${format} Foundation oracle omitted ${required.root} ${required.literal}`,
);
}
const archivePath = join(root, archive.filename);
const digest = createHash("sha256")
.update(await readFile(archivePath))
.digest("hex");
assert.equal(digest, archive.sha256, `${format} archive digest`);
archives.push({ format, archive, archivePath, digest });
}
return archives;
}
/** Check every oracle numeric slot, preserving object-table identity lookups. */
export function assertKeyedArchiveIntegerResult(
result,
format,
digest,
oracle,
) {
const parsed = keyedArchiveResultSchema.parse(result);
assert.equal(parsed.archive_sha256, digest);
assert.equal(parsed.target_sha256, digest);
assert.equal(
parsed.archive_format,
format === "xml" ? "xml-plist" : "binary-plist",
);
for (const row of oracle.rows) {
const unsafe = isUnsafeInteger(row);
const source =
row.container === "roots"
? parsed.roots
: parsed.objects.find((object) => object.id === row.objectId)?.value;
const value = valueAt(source, row.path);
assert.notEqual(
value,
undefined,
`${row.root} numeric slot remains present`,
);
if (unsafe) {
const exact =
typeof value === "object" &&
value !== null &&
value.$plist_type === "integer" &&
value.decimal === BigInt(row.literal).toString();
const disclosed = parsed.limitations.some(
(reason) =>
/integer|number/iu.test(reason) &&
/precision|exact range|rounded/iu.test(reason),
);
assert.ok(
exact || disclosed,
`${row.root} ${row.literal} must be exact or disclose numeric loss`,
);
} else {
assert.equal(
typeof value,
"number",
`${row.root} retains its numeric type`,
);
assert.equal(value, Number(row.literal), `${row.root} numeric value`);
}
}
return parsed;
}
+40 -22
View File
@@ -1,5 +1,5 @@
import { execFile, spawnSync } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { execFile } from "node:child_process";
import { mkdir, mkdtemp, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -11,6 +11,10 @@ import {
artifactMcpResult,
withArtifactMcp,
} from "../../lib/artifact-e2e.mjs";
import {
assertKeyedArchiveIntegerResult,
verifyKeyedArchiveIntegerArchives,
} from "./keyed-archive-integers.mjs";
if (process.platform !== "darwin")
throw new Error(
@@ -170,26 +174,37 @@ try {
);
});
}
const numericTests = spawnSync(
"npm",
[
"run",
"test:focused",
"--",
"tests/boundary/filesystem/keyedArchiveNativeIntegers.test.ts",
],
{
cwd: fileURLToPath(new URL("../../../", import.meta.url)),
stdio: "inherit",
timeout: 240_000,
},
);
if (numericTests.error) throw numericTests.error;
assert.equal(
numericTests.status,
0,
"Native Foundation integer cases failed",
);
const integers = join(root, "integers");
await mkdir(integers);
await promisify(execFile)("/usr/bin/xcrun", [
"swift",
"-module-cache-path",
join(root, "modules"),
fileURLToPath(
new URL(
"../../../tests/conformance/native/keyed-archive-integers.swift",
import.meta.url,
),
),
integers,
]);
for (const fixture of await verifyKeyedArchiveIntegerArchives(integers)) {
const cliResult = assertKeyedArchiveIntegerResult(
await artifactCli("inspect-keyed-archive", fixture.archivePath),
fixture.format,
fixture.digest,
fixture.archive,
);
await withArtifactMcp(fixture.archivePath, async (client) => {
const mcpResult = assertKeyedArchiveIntegerResult(
await artifactMcpResult(client, "inspect_keyed_archive"),
fixture.format,
fixture.digest,
fixture.archive,
);
assert.deepEqual(mcpResult, cliResult);
});
}
report = {
ok: true,
mocked: false,
@@ -197,6 +212,9 @@ try {
stdio_mcp: true,
xml_golden: true,
uid_named_root: true,
numeric_formats: ["xml", "binary"],
numeric_cli: true,
numeric_stdio_mcp: true,
format: graph.archive_format,
objects: graph.total_objects,
references: graph.total_references,
+62 -16
View File
@@ -42,11 +42,66 @@ try {
"",
].join("\n"),
);
const output = await runHeadless(target, "seed-regression", [
"-postScript",
"ReaSeedRegressionProbe.java",
seeds,
]);
// Headless can exit zero after a script fails, so require the probe's marker.
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
const analyzer = "Decompiler Switch Analysis";
const disabled = await runHeadless(target, "analyzer-regression-valid", [
"-preScript",
"ReaGhidraAnalysisOptions.java",
analyzer,
"-postScript",
"ReaAnalyzerOptionsRegressionProbe.java",
analyzer,
]);
assert.ok(
disabled.includes(`REA_ANALYZER_OPTIONS_DISABLED_OK ${analyzer}`),
disabled,
);
const unknownAnalyzer = "REA Unknown Analyzer";
const rejected = await runHeadless(target, "analyzer-regression-unknown", [
"-preScript",
"ReaGhidraAnalysisOptions.java",
`${analyzer},${unknownAnalyzer}`,
"-postScript",
"ReaAnalyzerOptionsRegressionProbe.java",
analyzer,
]);
assert.ok(
rejected.includes(
`Unknown Ghidra analyzer for this program: ${unknownAnalyzer}`,
),
rejected,
);
assert.ok(
rejected.includes("Processing aborted as a result of pre-script."),
rejected,
);
assert.ok(!rejected.includes("REA_ANALYZER_OPTIONS_DISABLED_OK"), rejected);
} finally {
if (supervisor !== undefined) {
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
}
await rm(workspace, { recursive: true, force: true });
}
console.log(
"Real Ghidra pre-script regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, analyzer option readback, unknown-analyzer abort before post-scripts, and owned-process cleanup.",
);
async function runHeadless(target, project, scriptArguments) {
const launch = await spawnOwnedProviderProcess({
command: join(installation, "support", "analyzeHeadless"),
arguments: [
workspace,
"seed-regression",
project,
"-import",
target,
"-loader",
@@ -63,9 +118,7 @@ try {
new URL("../../../tests/conformance/ghidra", import.meta.url),
),
].join(";"),
"-postScript",
"ReaSeedRegressionProbe.java",
seeds,
...scriptArguments,
],
runId: run.run_id,
expectedCommand: null,
@@ -82,16 +135,9 @@ try {
const snapshot = supervisor.snapshot();
const output = `${snapshot.stdout.text}\n${snapshot.stderr.text}`;
assert.equal(snapshot.exitCode, 0, output);
// Headless can exit zero after a script fails, so require the probe's marker.
assert.ok(output.includes("REA_SEED_REGRESSION_OK "), output);
} finally {
if (supervisor !== undefined) {
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
}
await rm(workspace, { recursive: true, force: true });
const stopped = await supervisor.stop();
assert.notEqual(stopped.status, "incomplete", JSON.stringify(stopped));
supervisor.dispose();
supervisor = undefined;
return output;
}
console.log(
"Real Ghidra seed regression passed: descending function order, out-of-range seed, BSS label, BSS code rejection, and owned-process cleanup.",
);
+41
View File
@@ -0,0 +1,41 @@
import { describe, expect, it } from "vitest";
import { adbHostFileName } from "./AdbHostFileName.js";
describe("adbHostFileName", () => {
it.each([
["/data/local/tmp/notes.txt", "notes.txt"],
["/data/data/com.example/databases/app.db", "app.db"],
["/sdcard/My Notes.txt", "My Notes.txt"],
["/system/etc/hosts", "hosts"],
[
"/data/local/tmp/split_config.arm64_v8a.apk",
"split_config.arm64_v8a.apk",
],
["/data/local/tmp/résumé.xml", "résumé.xml"],
["/data/local/tmp/a-b_c.d", "a-b_c.d"],
["/data/local/tmp/", "tmp"],
])("accepts the portable device path %s", (devicePath, fileName) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: true, fileName });
});
it.each([
["/", "empty_file_name"],
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/.", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/re\u007fport", "control_character"],
["/data/local/tmp/ta\tb", "control_character"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/nul.txt", "reserved_name"],
["/data/local/tmp/com1", "reserved_name"],
["/data/local/tmp/LPT9", "reserved_name"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report ", "trailing_dot_or_space"],
["/data/local/tmp/a*b.txt", "unsupported_character"],
["/data/local/tmp/a?b.txt", "unsupported_character"],
] as const)("refuses %s as %s", (devicePath, reason) => {
expect(adbHostFileName(devicePath)).toEqual({ ok: false, reason });
});
});
+61
View File
@@ -0,0 +1,61 @@
import { posix } from "node:path";
/** Why a device-reported path cannot be used as a host file name. */
export type AdbHostFileNameReason =
| "empty_file_name"
| "dot_segment"
| "path_separator"
| "control_character"
| "unsupported_character"
| "reserved_name"
| "trailing_dot_or_space"
| "alternate_data_stream";
/** A portable host file name, or the exact reason the device name was refused. */
export type AdbHostFileNameResult =
| { readonly ok: true; readonly fileName: string }
| { readonly ok: false; readonly reason: AdbHostFileNameReason };
/**
* Win32 resolves these basenames to a device regardless of case and of any
* extension (`NUL`, `nul.txt`), so a portable host file name must avoid them on
* every host. `(?:\.|$)` keeps ordinary names such as `console.txt` admissible.
*/
const WINDOWS_RESERVED_BASENAME =
/^(?:con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\.|$)/iu;
/** Characters that are not portable in a host file name on Windows or POSIX. */
const UNSUPPORTED_CHARACTER = /[*?"<>|]/u;
/** Control characters (`Cc`), including C0 controls and DEL. */
const CONTROL_CHARACTER = /\p{Cc}/u;
/**
* Derive a portable host file name from a device-reported path.
*
* The device path is a POSIX path on the device filesystem, so its last
* `/`-separated segment is taken with POSIX semantics rather than the host
* `basename` (which would vary between POSIX and Windows hosts). A name is
* accepted only when every host can store it and address it back unambiguously:
* no separator, control, or reserved-device characters, no trailing dot or
* space, no `:` alternate-data-stream marker, and no `.`/`..` segment.
*/
export const adbHostFileName = (devicePath: string): AdbHostFileNameResult => {
const fileName = posix.basename(devicePath);
if (fileName === "") return { ok: false, reason: "empty_file_name" };
if (fileName === "." || fileName === "..")
return { ok: false, reason: "dot_segment" };
if (fileName.includes("\\") || fileName.includes("/"))
return { ok: false, reason: "path_separator" };
if (CONTROL_CHARACTER.test(fileName))
return { ok: false, reason: "control_character" };
if (fileName.includes(":"))
return { ok: false, reason: "alternate_data_stream" };
if (UNSUPPORTED_CHARACTER.test(fileName))
return { ok: false, reason: "unsupported_character" };
if (WINDOWS_RESERVED_BASENAME.test(fileName))
return { ok: false, reason: "reserved_name" };
if (/[. ]$/u.test(fileName))
return { ok: false, reason: "trailing_dot_or_space" };
return { ok: true, fileName };
};
+68
View File
@@ -336,12 +336,38 @@ it("reports a missing selected binary as a capability gap with recovery", async
if (result.ok) return;
const projected = projectAnalysisError(result.error);
expect(projected).toMatchObject({ code: "capability_unavailable" });
expect(projected.message).toContain("REA_ADB_PATH");
expect(projected.remediation).toMatchObject({
action: expect.stringContaining("REA_ADB_PATH") as unknown,
});
expect(projected.details).toMatchObject({
provider_id: "adb",
reason: expect.stringContaining("not found") as unknown,
});
});
it("reports relative adb configuration as an actionable refusal", async () => {
const provider = new AdbProvider({
environment: { REA_ADB_PATH: "relative/adb" },
});
const availability = await provider.inspectAvailability();
expect(availability).toMatchObject({
status: "unavailable",
reason: expect.stringContaining("REA_ADB_PATH") as unknown,
});
const projected = await refusedBy(provider, {
operation: "list_adb_devices",
input: {},
});
expect(projected).toMatchObject({
code: "capability_unavailable",
message: expect.stringContaining("REA_ADB_PATH") as unknown,
remediation: {
action: expect.stringContaining("REA_ADB_PATH") as unknown,
},
});
});
it("cancels before spawning anything", async () => {
const { binary, root } = await writeAdbStub(
(root) => `: > ${join(root, "spawned")}
@@ -452,6 +478,48 @@ exit 1`,
});
});
it.each([
["/data/local/tmp/..", "dot_segment"],
["/data/local/tmp/a\\b", "path_separator"],
["/data/local/tmp/CON", "reserved_name"],
["/data/local/tmp/report.", "trailing_dot_or_space"],
["/data/local/tmp/report:2024", "alternate_data_stream"],
] as const)(
"refuses the unsafe device path %s before pulling",
async (devicePath, reason) => {
const { binary, root } = await writeAdbStub(
(root) => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
: > "${join(root, "pull-ran")}"
exit 0`,
);
const output = join(root, "pulls");
const result = await providerOver(binary).execute({
operation: "pull_adb_file",
input: {
serial: "emulator-5554",
device_path: devicePath,
output_directory: output,
},
});
expect(result.ok).toBe(false);
if (result.ok) return;
const projected = projectAnalysisError(result.error);
expect(projected).toMatchObject({ code: "invalid_request" });
expect(projected.details).toMatchObject({
issues: [
{
path: ["device_path"],
reason: "invalid_value",
message: expect.stringContaining(reason),
},
],
});
// No host directory and no adb invocation are effects of a refused pull.
await expect(readFile(join(root, "pull-ran"))).rejects.toThrow();
await expect(readFile(output)).rejects.toThrow();
},
);
it("pushes a file and verifies the device-side digest", async () => {
const { binary, root } = await writeAdbStub(
() => `if [ "$1" = "version" ]; then ${VERSION_BLOCK}; exit 0; fi
+20 -11
View File
@@ -2,12 +2,13 @@ import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import { constants as fsConstants } from "node:fs";
import { access, lstat, mkdir, rm, stat, writeFile } from "node:fs/promises";
import { basename, join, resolve } from "node:path";
import { join, resolve } from "node:path";
import type {
ExecutionOptions,
ProviderAvailability,
} from "../application/AnalysisProvider.js";
import { providerConfigurationRefusal } from "../domain/providerConfigurationRefusal.js";
import {
createAnalysisExecution,
type AnalysisExecution,
@@ -15,7 +16,6 @@ import {
import type { AdbRequest } from "../domain/adb/adbDeviceAnalysis.js";
import {
AnalysisArtifactChangedError,
AnalysisCapabilityUnavailableError,
AnalysisCancelledError,
AnalysisInputError,
AnalysisOutputError,
@@ -47,6 +47,7 @@ import {
parsePackageListOutput,
parsePmPathOutput,
} from "./AdbDeviceOutput.js";
import { adbHostFileName } from "./AdbHostFileName.js";
import {
parseAmStartOutput,
parseBugreportCopiedPath,
@@ -417,8 +418,8 @@ export class AdbProvider {
async inspectAvailability(
signal?: AbortSignal,
): Promise<ProviderAvailability> {
const selection = this.selection();
try {
const selection = this.selection();
await inspectAdbClient(selection, signal);
return {
status: "available",
@@ -555,21 +556,19 @@ export class AdbProvider {
const signal = options?.signal;
if (signal?.aborted === true)
return err(new AnalysisCancelledError(request.operation));
const selection = this.selection();
let selection: AdbBinarySelection;
let identity: AdbClientIdentity;
try {
selection = this.selection();
identity = await inspectAdbClient(selection, signal);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) return err(cause);
if (cause instanceof AdbConfigurationFailure)
return err(
new AnalysisCapabilityUnavailableError(
providerConfigurationRefusal(
ADB_PROVIDER_IDENTITY.id,
request.operation,
typeof cause.diagnostics.remediation === "string"
? `${cause.message} ${cause.diagnostics.remediation}`
: cause.message,
{ cause },
cause,
),
);
return err(
@@ -1401,13 +1400,23 @@ export class AdbProvider {
},
signal: AbortSignal | undefined,
): Promise<Result<AnalysisExecution, AnalysisError>> {
const derived = adbHostFileName(input.device_path);
if (!derived.ok)
return err(
new AnalysisInputError(operation, undefined, [
{
path: ["device_path"],
reason: "invalid_value",
message: `The device path does not name a portable host file (${derived.reason}); REA never renames a device-reported file.`,
},
]),
);
const directory = await prepareOutputDirectory(
operation,
input.output_directory,
);
if (!directory.ok) return directory;
const fileName = basename(input.device_path) || "pulled";
const localPath = join(directory.value, fileName);
const localPath = join(directory.value, derived.fileName);
const existing = await refuseExistingTarget(operation, localPath);
if (existing !== undefined) return err(existing);
const pulled = await runAdb(
+11 -10
View File
@@ -20,6 +20,7 @@ import type {
ExecutionOptions,
ProviderAvailability,
} from "../application/AnalysisProvider.js";
import { providerConfigurationRefusal } from "../domain/providerConfigurationRefusal.js";
import {
createAnalysisExecution,
type AnalysisExecution,
@@ -153,12 +154,13 @@ const runApktool = async (
invocation = apktoolInvocation(selection, arguments_, hostEnvironment);
} catch (cause) {
if (!(cause instanceof WindowsBatchCommandError)) throw cause;
const reason = `The apktool batch launcher cannot receive REA's decode paths through cmd.exe: ${cause.message}`;
return err(
new AnalysisCapabilityUnavailableError(
APKTOOL_PROVIDER_IDENTITY.id,
operation,
`The apktool batch launcher cannot receive REA's decode paths through cmd.exe: ${cause.message}`,
{ cause },
reason,
{ cause, userMessage: reason },
),
);
}
@@ -345,8 +347,8 @@ export class ApktoolProvider {
async #inspectAvailability(
signal: AbortSignal,
): Promise<ProviderAvailability> {
const selection = this.selection();
try {
const selection = this.selection();
await inspectApktoolClient(selection, signal, this.environment);
throwIfCancelled("inspect_apktool_client", signal);
return {
@@ -397,9 +399,10 @@ export class ApktoolProvider {
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (signal?.aborted === true)
return err(new AnalysisCancelledError(request.operation));
const selection = this.selection();
let selection: ApktoolCommandSelection;
let identity: ApktoolClientIdentity;
try {
selection = this.selection();
identity = await inspectApktoolClient(
selection,
signal,
@@ -407,17 +410,15 @@ export class ApktoolProvider {
);
} catch (cause) {
if (cause instanceof AnalysisCancelledError) return err(cause);
if (cause instanceof ApktoolConfigurationFailure)
if (cause instanceof ApktoolConfigurationFailure) {
return err(
new AnalysisCapabilityUnavailableError(
providerConfigurationRefusal(
APKTOOL_PROVIDER_IDENTITY.id,
request.operation,
typeof cause.diagnostics.remediation === "string"
? `${cause.message} ${cause.diagnostics.remediation}`
: cause.message,
{ cause },
cause,
),
);
}
return err(
new AnalysisProtocolError(
`apktool version probe failed for ${request.operation}`,
+21 -3
View File
@@ -1,4 +1,5 @@
import type {
AnalysisExecution,
AnalysisOperation,
ExecutionOptions,
} from "./AnalysisProvider.js";
@@ -17,6 +18,13 @@ type AnalysisCall = (
selectorOrigin?: ExecutionOptions["selectorOrigin"],
) => Promise<Result<JsonValue, AnalysisError>>;
type RelationshipCall = (
name: AnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
) => Promise<Result<AnalysisExecution, AnalysisError>>;
export interface CallPathTraceInput {
readonly start: string;
readonly goal?: string | undefined;
@@ -42,6 +50,7 @@ interface TraceState {
readonly error: ReturnType<typeof projectAnalysisError>;
}>;
readonly residual: Set<string>;
readonly limitations: Set<string>;
queueIndex: number;
traversalPath: string[];
}
@@ -63,6 +72,7 @@ interface DiscoveredEdge {
/** Trace direct caller/callee relationships until the graph is exhausted or the goal is reached. */
export const traceCallPath = async (
call: AnalysisCall,
callRelationship: RelationshipCall,
input: CallPathTraceInput,
signal?: AbortSignal,
): EnhancedResult => {
@@ -88,7 +98,7 @@ export const traceCallPath = async (
if (address === undefined) break;
const current = state.visited.get(address);
if (current === undefined) continue;
const found = await expandAddress(call, {
const found = await expandAddress(callRelationship, {
input: resolvedInput,
state,
address,
@@ -106,12 +116,13 @@ const createTraceState = (input: CallPathTraceInput): TraceState => ({
edges: new Map(),
failures: [],
residual: new Set(),
limitations: new Set(),
queueIndex: 0,
traversalPath: input.goal === input.start ? [input.start] : [],
});
const expandAddress = async (
call: AnalysisCall,
call: RelationshipCall,
expansion: AddressExpansion,
): Promise<boolean> => {
const { input, state, address, current, signal } = expansion;
@@ -127,7 +138,7 @@ const expandAddress = async (
state.residual.add(`Call relationships were unavailable for ${address}.`);
return false;
}
const related = parseRelatedAddresses(result.value, relation);
const related = parseRelatedAddresses(result.value.result, relation);
if (!related.ok) {
state.failures.push({
address,
@@ -136,6 +147,12 @@ const expandAddress = async (
state.residual.add(`Call relationships were unreadable for ${address}.`);
return false;
}
for (const limitation of result.value.limitations) {
state.limitations.add(limitation);
state.residual.add(
`Call relationships for ${address} are limited: ${limitation}`,
);
}
for (const relatedAddress of sortedUniqueAddresses(related.value)) {
recordEdge(input, state, {
address,
@@ -215,6 +232,7 @@ const projectTrace = (input: CallPathTraceInput, state: TraceState) => ({
residual_unknowns: [...state.residual],
limitations: [
"Direct provider call relationships may omit unresolved indirect calls.",
...state.limitations,
],
});
+36
View File
@@ -281,6 +281,42 @@ describe("doctor installation identity", () => {
});
describe("doctor scoped readiness", () => {
it("checks explicitly selected skill destinations independently of registrations", async () => {
const selected: (readonly string[] | undefined)[] = [];
const result = await runDoctor(
undefined,
host({
clientRegistrations: () => Promise.resolve([]),
installedSkillIdentity: (clients) => {
selected.push(clients);
return Promise.resolve({
version: "stale",
toolCount: 0,
canonical: false,
});
},
}),
{
clients: [],
skillClients: ["claude_code", "claude_code"],
skill: true,
},
);
expect(selected).toEqual([["claude_code"]]);
expect(result.scope).toMatchObject({
mode: "explicit",
clients: [],
skill_clients: ["claude_code"],
skill: true,
});
expect(result.healthy).toBe(false);
expect(result.scope_checks.map(({ name }) => name)).toEqual([
"node",
"skill:identity",
]);
});
it("scopes effective readiness to selected clients without hiding drift", async () => {
const registrations: readonly ClientRegistrationStatus[] = [
{
+4 -2
View File
@@ -120,6 +120,7 @@ export interface DoctorReport {
/** Caller-selected readiness boundary; omitted values retain audit semantics. */
export interface DoctorScope {
readonly clients?: readonly string[];
readonly skillClients?: readonly string[];
readonly providers?: readonly string[];
readonly skill?: boolean;
}
@@ -206,9 +207,10 @@ const collectDoctorIdentity = async (
const installationPaths = (await host.installationPaths?.()) ?? [];
const registrations = (await host.clientRegistrations?.()) ?? [];
const skillClientIds =
scope.mode === "audit-wide"
scope.skill_clients ??
(scope.mode === "audit-wide"
? registrations.map(({ client }) => client)
: scope.clients;
: scope.clients);
const installedSkillIdentity =
await host.installedSkillIdentity?.(skillClientIds);
const skillAligned = skillIdentityAligned(installedSkillIdentity);
+4
View File
@@ -14,6 +14,7 @@ import type {
export interface DoctorScopeReport {
readonly mode: "audit-wide" | "explicit";
readonly clients: readonly string[];
readonly skill_clients?: readonly string[];
readonly providers: readonly string[];
readonly skill: boolean;
readonly target: string | null;
@@ -40,6 +41,9 @@ export const normalizeDoctorScope = (
): DoctorScopeReport => ({
mode: requested === undefined ? "audit-wide" : "explicit",
clients: uniqueSorted(requested?.clients ?? []),
...(requested?.skillClients === undefined
? {}
: { skill_clients: uniqueSorted(requested.skillClients) }),
providers: uniqueSorted(requested?.providers ?? []),
skill: requested?.skill === true,
target: target ?? null,
+24 -3
View File
@@ -1,6 +1,7 @@
import { z } from "zod";
import type {
AnalysisExecution,
AnalysisOperation,
AnalysisOperationPort,
ExecutionOptions,
@@ -166,6 +167,8 @@ export class EnhancedTools {
return traceCallPath(
(name, arguments_, operationSignal, selectorOrigin) =>
this.#call(name, arguments_, operationSignal, selectorOrigin),
(name, arguments_, operationSignal, selectorOrigin) =>
this.#observe(name, arguments_, operationSignal, selectorOrigin),
call.input,
signal,
);
@@ -359,7 +362,7 @@ export class EnhancedTools {
const level = String(current.depth);
graph[level] ??= [];
const result = await this.#call(
const result = await this.#observe(
tool,
{ procedure: current.address },
signal,
@@ -373,7 +376,7 @@ export class EnhancedTools {
});
continue;
}
const related = parseRelatedAddresses(result.value, relation);
const related = parseRelatedAddresses(result.value.result, relation);
if (!related.ok) {
graph[level].push({
address: current.address,
@@ -386,6 +389,9 @@ export class EnhancedTools {
address: current.address,
status: "ok",
calls: [...related.value],
...(result.value.limitations.length === 0
? {}
: { limitations: [...result.value.limitations] }),
});
for (const address of related.value) {
if (!discovered.has(address)) {
@@ -523,13 +529,28 @@ export class EnhancedTools {
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
): Promise<Result<JsonValue, AnalysisError>> {
const execution = await this.#observe(
name,
arguments_,
signal,
selectorOrigin,
);
return execution.ok ? ok(execution.value.result) : execution;
}
async #observe(
name: AnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
signal?: AbortSignal,
selectorOrigin?: ExecutionOptions["selectorOrigin"],
): Promise<Result<AnalysisExecution, AnalysisError>> {
if (isAborted(signal)) return err(new AnalysisCancelledError(name));
const execution = await this.analysis.execute(name, arguments_, {
...(signal === undefined ? {} : { signal }),
...(selectorOrigin === undefined ? {} : { selectorOrigin }),
});
if (isAborted(signal)) return err(new AnalysisCancelledError(name));
return execution.ok ? ok(execution.value.result) : execution;
return execution;
}
}
@@ -18,6 +18,7 @@ import type {
ReferenceSourceRead,
} from "../reference/ReferenceSourceReaderTypes.js";
import { PARSEABLE_REFERENCE_SOURCE_LANGUAGES } from "./ReferenceSourceImportTypes.js";
import { typescriptEmittedSourceCandidates } from "./TypeScriptEmittedSourceCandidates.js";
export interface ParsedReferenceSourceEntries {
readonly entries: HistoricalSourceGraphInput["entries"];
@@ -99,22 +100,9 @@ const resolveInternalSpecifier = (
`${normalized}${suffix}`,
`${normalized}/index${suffix}`,
]);
// TypeScript source commonly names the extension emitted for NodeNext.
// Retain present runtime files first; substitute only a missing counterpart.
// Counterparts cover explicit filenames, not directory/index resolution.
if (/\.(?:ts|tsx|mts|cts)$/u.test(fromPath)) {
const sourceSuffixes = normalized.endsWith(".js")
? [".ts", ".tsx", ".d.ts"]
: normalized.endsWith(".mjs")
? [".mts", ".d.mts"]
: normalized.endsWith(".cjs")
? [".cts", ".d.cts"]
: normalized.endsWith(".jsx")
? [".tsx", ".d.ts"]
: [];
const stem = normalized.slice(0, -posix.extname(normalized).length);
candidates.push(...sourceSuffixes.map((suffix) => `${stem}${suffix}`));
}
candidates.push(...typescriptEmittedSourceCandidates(fromPath, normalized));
const match = lookupPath.resolveCandidates(candidates);
return match === undefined
? { to: normalized, resolution: "unresolved" }
+21 -2
View File
@@ -8,6 +8,8 @@ import type {
} from "./SetupTypes.js";
import type { SetupClient } from "./SupportedClients.js";
import type { SetupHopperInstallResult } from "./SetupInstallFailure.js";
import { err, ok } from "../domain/result.js";
import { SetupSkillInstallError } from "./SetupSkill.js";
import type { DoctorCheck, DoctorReport, DoctorScope } from "./Doctor.js";
import type { LinuxDistribution } from "./LinuxHopper.js";
import { PRODUCT_IDENTITY, SDK_IDENTITY } from "../identity.js";
@@ -122,9 +124,22 @@ export class FakeSetupHost implements SetupHost {
);
skillNeedsInstall = (): Promise<boolean> =>
Promise.resolve(this.skill !== "unchanged");
installSkill = (): Promise<"installed" | "unchanged" | "failed"> => {
installSkill = () => {
this.skillInstalls += 1;
return Promise.resolve(this.skill);
return Promise.resolve(
this.skill === "failed"
? err(
new SetupSkillInstallError({
path: "/fixture/home/.agents/skills/reverse-engineer-anything/SKILL.md",
stage: "write",
reason: "injected skill write failure",
rollbackFailures: [],
recoveryBackupPaths: [],
cause: new Error("injected skill write failure"),
}),
)
: ok(this.skill),
);
};
doctor = (scope?: DoctorScope): Promise<DoctorReport> => {
this.doctorCalls += 1;
@@ -159,6 +174,7 @@ export class FakeSetupHost implements SetupHost {
scope === undefined
? environmentHealthy
: (scope.clients?.length ?? 0) === 0 &&
(scope.skillClients?.length ?? 0) === 0 &&
(scope.providers?.length ?? 0) === 0 &&
scope.skill !== true
? true
@@ -169,6 +185,9 @@ export class FakeSetupHost implements SetupHost {
scope: {
mode: scope === undefined ? "audit-wide" : "explicit",
clients: scope?.clients ?? [],
...(scope?.skillClients === undefined
? {}
: { skill_clients: scope.skillClients }),
providers: scope?.providers ?? [],
skill: scope?.skill === true,
target: null,
+19 -4
View File
@@ -6,6 +6,7 @@ import { ConfigurationError } from "../domain/configurationErrors.js";
import { SUPPORTED_NODE_VERSION_PROSE } from "../domain/runtimeVersion.js";
import { FakeSetupHost, options } from "./Setup.fixture.js";
import { runSetup } from "./Setup.js";
import type { SetupProgressEvent } from "./SetupTypes.js";
describe("setup workflow", () => {
it("rejects Node 25 before applying setup actions", async () => {
@@ -57,14 +58,27 @@ describe("setup workflow", () => {
const host = new FakeSetupHost();
host.hopper = "/Applications/Hopper";
host.skill = "failed";
const progress: SetupProgressEvent[] = [];
const result = await runSetup(
{ ...options(true), installSkill: true, clientIds: [] },
{
...options(true),
installSkill: true,
clientIds: [],
onProgress: (event) => {
if (event.actionId === "install_skill") progress.push(event);
},
},
host,
);
expect(result.status).toBe("needs_human");
expect(result.remediation).toBe(
"REA analysis skill could not be installed or verified. Check permissions for its planned skill directories, then rerun setup.",
);
expect(result.code).toBe("skill_install_failed");
expect(result.remediation).toContain("REA skill write failed at");
expect(result.remediation).toContain("injected skill write failure");
expect(result.remediation).toContain("Check permissions and rerun setup.");
expect(progress.at(-1)).toMatchObject({
state: "failed",
detail: expect.stringContaining("injected skill write failure"),
});
});
it("delegates remaining unhealthy checks to doctor remediation", async () => {
@@ -232,6 +246,7 @@ describe("setup scoped readiness", () => {
clients: ["codex"],
providers: [],
skill: true,
skillClients: ["codex"],
});
});
+27 -15
View File
@@ -26,6 +26,7 @@ import {
import { AnalysisError } from "../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../domain/analysisErrorCleanup.js";
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
import type { SetupSkillInstallError } from "./SetupSkill.js";
/** Discover, approve, and apply setup actions idempotently. */
export const runSetup = async (
@@ -125,7 +126,10 @@ const runSetupWorkflow = async (
: (options.clientIds ??
options.readinessScope?.clients ??
discovery.defaultClientIds);
let selectedSkillClientIds = options.skillClientIds ?? selectedClientIds;
let selectedSkillClientIds =
options.skillClientIds ??
options.readinessScope?.skillClients ??
selectedClientIds;
let skillNeedsInstall = await host.skillNeedsInstall(selectedSkillClientIds);
let skillSelected =
options.installSkill === true ||
@@ -220,6 +224,7 @@ const runSetupWorkflow = async (
installHopper: planDiscovery.installHopper,
installSkill,
skillSelected,
selectedSkillClientIds,
selectedClients: planned.selectedClients,
clientsToConfigure: planned.clientsToConfigure,
};
@@ -311,18 +316,20 @@ const runSetupWorkflow = async (
: { onProgress: options.onProgress }),
});
if (clientFailure !== undefined) return fail(clientFailure);
if (
planSelection.installSkill &&
!(await installSkillAction(
if (planSelection.installSkill) {
const skillFailure = await installSkillAction(
host,
options,
appliedActions,
selectedSkillClientIds,
))
)
return fail(
"REA analysis skill could not be installed or verified. Check permissions for its planned skill directories, then rerun setup.",
);
if (skillFailure !== undefined) {
return {
...(await fail(skillFailure.remediation)),
code: "skill_install_failed",
};
}
}
const doctor = summarizeDoctor(await host.doctor(readinessScope));
const remediation = combineRemediation(
finalSetupRemediation(
@@ -413,7 +420,7 @@ const installSkillAction = async (
options: SetupOptions,
appliedActions: string[],
skillClientIds: readonly string[],
): Promise<boolean> => {
): Promise<SetupSkillInstallError | undefined> => {
const label = "REA reverse-engineering skill";
emitProgress(options, {
actionId: "install_skill",
@@ -421,22 +428,23 @@ const installSkillAction = async (
state: "started",
});
const skill = await host.installSkill(skillClientIds);
if (skill === "failed") {
if (!skill.ok) {
emitProgress(options, {
actionId: "install_skill",
label,
state: "failed",
detail: skill.error.message,
});
return false;
return skill.error;
}
if (skill === "installed") appliedActions.push("installed_skill");
if (skill.value === "installed") appliedActions.push("installed_skill");
emitProgress(options, {
actionId: "install_skill",
label,
state: skill === "installed" ? "completed" : "warning",
...(skill === "unchanged" ? { detail: "Already current" } : {}),
state: skill.value === "installed" ? "completed" : "warning",
...(skill.value === "unchanged" ? { detail: "Already current" } : {}),
});
return true;
return undefined;
};
const setupFailure = async (
@@ -519,6 +527,7 @@ const resolvedReadinessScope = (
readonly selectedClients: readonly SetupClient[];
readonly installHopper: boolean;
readonly skillSelected?: boolean;
readonly selectedSkillClientIds: readonly string[];
},
): DoctorScope => ({
clients: selection.selectedClients.map(({ name }) => name),
@@ -526,6 +535,9 @@ const resolvedReadinessScope = (
? ["hopper"]
: (requested?.providers ?? []),
skill: selection.skillSelected === true || requested?.skill === true,
...(selection.skillSelected === true || requested?.skill === true
? { skillClients: selection.selectedSkillClientIds }
: {}),
});
const summarizeDoctor = (report: DoctorReport): SetupDoctorSummary => ({
+1
View File
@@ -11,6 +11,7 @@ export type SetupFailureCode =
| "mount_failed"
| "bundle_invalid"
| "copy_failed"
| "skill_install_failed"
| "cleanup_failed";
export type SetupHopperInstallResult =
+183 -46
View File
@@ -2,10 +2,15 @@ import { copyFile, lstat, mkdir, readFile, rm } from "node:fs/promises";
import { constants as fsConstants } from "node:fs";
import { dirname, join } from "node:path";
import writeFileAtomic from "write-file-atomic";
import { err, ok, type Result } from "../domain/result.js";
import { readBoundedFileBytes } from "../process/BoundedFileBytes.js";
import { PRODUCT_IDENTITY } from "../identity.js";
import { NonRegularFileReadError } from "../filesystem/RegularFile.js";
import { readRegularFileText } from "./RegularFileRead.js";
import {
NonRegularFileReadError,
RegularFileChangedError,
} from "../filesystem/RegularFile.js";
import { withRegularFile } from "./RegularFileRead.js";
import { clientSkillDirectories } from "./SupportedClients.js";
export const MANAGED_SKILL_FILES = [
@@ -23,14 +28,119 @@ export const isManagedSkillManifest = (content: string): boolean =>
content.startsWith(`---\r\nname: ${PRODUCT_IDENTITY.skillName}\r\n`);
export const readSkillFile = (path: string): Promise<string> =>
readRegularFileText(path, { symlinks: "reject" });
readSkillFileState(path).then(({ content }) => content);
interface SkillFileState {
readonly content: string;
readonly mode: number;
}
const readSkillFileState = async (path: string): Promise<SkillFileState> =>
withRegularFile(
path,
async (handle, stats) => {
const bytes = await readBoundedFileBytes(
handle,
stats.size,
undefined,
stats.size,
);
if (bytes === undefined || bytes.length !== stats.size)
throw new RegularFileChangedError(path);
return { content: bytes.toString("utf8"), mode: stats.mode & 0o7777 };
},
{ symlinks: "reject" },
);
interface CanonicalSkillFile {
readonly content: string;
readonly destination: string;
readonly original: string | undefined;
readonly original: SkillFileState | undefined;
}
export interface SkillInstallFileOperations {
readonly mkdir: (path: string) => Promise<void>;
readonly backup: (source: string, destination: string) => Promise<void>;
readonly write: (
path: string,
content: string,
mode: number,
) => Promise<void>;
readonly remove: (path: string) => Promise<void>;
}
export interface SkillInstallRollbackFailure {
readonly path: string;
readonly reason: string;
}
/** Expected failure while replacing a managed skill bundle. */
export class SetupSkillInstallError extends Error {
readonly _tag = "SetupSkillInstallError";
readonly path: string;
readonly stage: "prepare" | "backup" | "write" | "readback";
readonly reason: string;
readonly rollbackFailures: readonly SkillInstallRollbackFailure[];
readonly recoveryBackupPaths: readonly string[];
constructor(input: {
readonly path: string;
readonly stage: "prepare" | "backup" | "write" | "readback";
readonly reason: string;
readonly rollbackFailures: readonly SkillInstallRollbackFailure[];
readonly recoveryBackupPaths: readonly string[];
readonly cause: unknown;
}) {
const message = [
`REA skill ${input.stage} failed at ${input.path}: ${input.reason}.`,
...(input.rollbackFailures.length === 0
? []
: [
`Rollback also failed for ${input.rollbackFailures.map(({ path, reason }) => `${path} (${reason})`).join(", ")}.`,
...(input.recoveryBackupPaths.length === 0
? []
: [
`Existing-file backups remain at ${input.recoveryBackupPaths.join(", ")}.`,
]),
]),
].join(" ");
super(message, { cause: input.cause });
this.path = input.path;
this.stage = input.stage;
this.reason = input.reason;
this.rollbackFailures = input.rollbackFailures;
this.recoveryBackupPaths = input.recoveryBackupPaths;
}
get remediation(): string {
return `${this.message} Check permissions and rerun setup.`;
}
}
export type SkillInstallValue = "installed" | "unchanged";
export type SkillInstallResult = Result<
SkillInstallValue,
SetupSkillInstallError
>;
const systemSkillInstallFileOperations: SkillInstallFileOperations = {
mkdir: async (path) => {
await mkdir(path, { recursive: true });
},
backup: async (source, destination) => {
try {
await copyFile(source, destination, fsConstants.COPYFILE_EXCL);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return;
throw cause;
}
},
write: (path, content, mode) =>
writeFileAtomic(path, content, { encoding: "utf8", mode }),
remove: (path) => rm(path, { force: true }),
};
/** Managed skill location selected for a client family. */
export interface SkillDestination {
readonly client: string;
@@ -87,9 +197,11 @@ export const existingSkillDestinations = async (
return existing;
};
const readOptionalText = async (path: string): Promise<string | undefined> => {
const readOptionalSkillState = async (
path: string,
): Promise<SkillFileState | undefined> => {
try {
return await readSkillFile(path);
return await readSkillFileState(path);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "ENOENT")
return undefined;
@@ -117,7 +229,7 @@ const canonicalSkillFiles = async (
),
"utf8",
),
original: await readOptionalText(destination),
original: await readOptionalSkillState(destination),
};
}),
),
@@ -133,7 +245,7 @@ export const canonicalSkillNeedsInstall = async (
try {
return (
await canonicalSkillFiles(home, clientIds, environment, platform)
).some(({ content, original }) => original !== content);
).some(({ content, original }) => original?.content !== content);
} catch (cause: unknown) {
// Unreadable skill state fails open to install so setup can repair it.
void cause;
@@ -141,39 +253,42 @@ export const canonicalSkillNeedsInstall = async (
}
};
const writeText = (path: string, content: string): Promise<void> =>
writeFileAtomic(path, content, { encoding: "utf8", mode: 0o600 });
const causeReason = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const preserveSkillBackup = async (
source: string,
destination: string,
): Promise<void> => {
try {
await copyFile(source, destination, fsConstants.COPYFILE_EXCL);
} catch (cause: unknown) {
if (cause instanceof Error && "code" in cause && cause.code === "EEXIST")
return;
throw cause;
}
};
const causePath = (cause: unknown): string | undefined =>
cause instanceof Error && "path" in cause && typeof cause.path === "string"
? cause.path
: undefined;
const restoreSkillFiles = async (
changed: readonly CanonicalSkillFile[],
): Promise<void> => {
for (const { destination, original } of [...changed].reverse()) {
if (original === undefined) await rm(destination, { force: true });
else await writeText(destination, original);
attempted: readonly CanonicalSkillFile[],
operations: SkillInstallFileOperations,
): Promise<readonly SkillInstallRollbackFailure[]> => {
const failures: SkillInstallRollbackFailure[] = [];
for (const { destination, original } of [...attempted].reverse()) {
try {
if (original === undefined) await operations.remove(destination);
else await operations.write(destination, original.content, original.mode);
} catch (cause: unknown) {
failures.push({ path: destination, reason: causeReason(cause) });
}
}
return failures;
};
/** Transactionally install or upgrade the canonical REA skill and references. */
/** Transactionally install the canonical skill bundle and optional references. */
export const installCanonicalSkill = async (
home: string,
clientIds: readonly string[] = [],
environment: Readonly<NodeJS.ProcessEnv> = {},
platform: NodeJS.Platform = process.platform,
): Promise<"installed" | "unchanged" | "failed"> => {
let changed: readonly CanonicalSkillFile[] = [];
operations: SkillInstallFileOperations = systemSkillInstallFileOperations,
): Promise<SkillInstallResult> => {
let attempted: CanonicalSkillFile[] = [];
let stage: "prepare" | "backup" | "write" | "readback" = "prepare";
let failingPath = "skill bundle";
try {
const canonical = await canonicalSkillFiles(
home,
@@ -181,30 +296,52 @@ export const installCanonicalSkill = async (
environment,
platform,
);
changed = canonical.filter(({ content, original }) => original !== content);
if (changed.length === 0) return "unchanged";
const changed = canonical.filter(
({ content, original }) => original?.content !== content,
);
if (changed.length === 0) return ok("unchanged");
stage = "backup";
for (const { destination, original } of changed) {
await mkdir(dirname(destination), { recursive: true });
failingPath = destination;
await operations.mkdir(dirname(destination));
if (original !== undefined)
await preserveSkillBackup(destination, `${destination}.rea.backup`);
await operations.backup(destination, `${destination}.rea.backup`);
}
for (const { destination, content } of changed)
await writeText(destination, content);
for (const { destination, content } of changed)
stage = "write";
for (const file of changed) {
failingPath = file.destination;
// An atomic writer can publish and then reject; include the target before
// entering it so rollback covers that ambiguous outcome.
attempted.push(file);
await operations.write(
file.destination,
file.content,
file.original?.mode ?? 0o600,
);
}
stage = "readback";
for (const { destination, content } of changed) {
failingPath = destination;
if ((await readSkillFile(destination)) !== content)
throw new Error(`skill readback mismatch: ${destination}`);
return "installed";
} catch (cause: unknown) {
// Install failure preserves the original error outcome; report cause inline.
void cause;
try {
await restoreSkillFiles(changed);
} catch (restoreCause: unknown) {
// Per-file backups remain beside changed files for operator recovery.
void restoreCause;
}
return "failed";
return ok("installed");
} catch (cause: unknown) {
failingPath = causePath(cause) ?? failingPath;
const rollbackFailures = await restoreSkillFiles(attempted, operations);
return err(
new SetupSkillInstallError({
path: failingPath,
stage,
reason: causeReason(cause),
rollbackFailures,
recoveryBackupPaths: attempted.flatMap(({ destination, original }) =>
original === undefined ? [] : [`${destination}.rea.backup`],
),
cause,
}),
);
}
};
+2 -4
View File
@@ -1,6 +1,6 @@
import type { DoctorReport, DoctorScope, runDoctor } from "./Doctor.js";
import type { LinuxDistribution } from "./LinuxHopper.js";
import type { SkillDestination } from "./SetupSkill.js";
import type { SkillDestination, SkillInstallResult } from "./SetupSkill.js";
import type { SetupClient } from "./SupportedClients.js";
import type {
SetupFailureCode,
@@ -70,9 +70,7 @@ export interface SetupHost {
command: readonly string[],
): Promise<ClientConfigurationInspection>;
skillNeedsInstall(clientIds: readonly string[]): Promise<boolean>;
installSkill(
clientIds: readonly string[],
): Promise<"installed" | "unchanged" | "failed">;
installSkill(clientIds: readonly string[]): Promise<SkillInstallResult>;
doctor(scope?: DoctorScope): Promise<Awaited<ReturnType<typeof runDoctor>>>;
}
/** Structured setup outcome carrying remediation instead of prompting. */
@@ -0,0 +1,29 @@
/** Emitted NodeNext suffixes and the TypeScript source counterparts they name. */
const EMITTED_SUFFIXES: ReadonlyArray<
readonly [suffix: string, sourceSuffixes: readonly string[]]
> = [
[".js", [".ts", ".tsx", ".d.ts"]],
[".mjs", [".mts", ".d.mts"]],
[".cjs", [".cts", ".d.cts"]],
[".jsx", [".tsx", ".d.ts"]],
];
/**
* Return TypeScript source counterparts for an explicit emitted filename.
* Callers retain exact runtime-file precedence and must not use these for
* directory/index, URL, HTML, or package target resolution.
*/
export const typescriptEmittedSourceCandidates = (
sourcePath: string,
candidate: string,
): readonly string[] => {
if (!/\.(?:ts|tsx|mts|cts)$/u.test(sourcePath)) return [];
const emitted = EMITTED_SUFFIXES.find(([suffix]) =>
candidate.endsWith(suffix),
);
if (emitted === undefined) return [];
// Strip the matched suffix by length. posix.extname would miss dotfile
// names such as ".js", whose whole basename is the emitted suffix itself.
const stem = candidate.slice(0, candidate.length - emitted[0].length);
return emitted[1].map((suffix) => `${stem}${suffix}`);
};
+1
View File
@@ -38,6 +38,7 @@ export interface WebModuleResolutionBatch {
/** Replaceable native resolution engine, with no provider protocol in callers. */
export interface WebModuleResolutionPort {
close?(): Promise<void>;
resolve(
input: {
readonly importerUrl: string;
+4
View File
@@ -69,6 +69,10 @@ export class WebModuleTraceService {
readonly resolver: WebModuleResolutionPort,
) {}
close(): Promise<void> {
return this.resolver.close?.() ?? Promise.resolve();
}
async trace(
rawInput: unknown,
options?: ExecutionOptions,
@@ -115,6 +115,63 @@ it("exports independent nested query, workflow and Evidence payloads", () => {
expect(serializeAnalysisSnapshot(fresh.value)).toBe(expected);
});
it("rebinds replayed target subjects while preserving child artifact subjects", () => {
const cache = new AnalysisSnapshotCache();
const target = ANALYSIS_SNAPSHOT_TARGET;
const profile = ANALYSIS_SNAPSHOT_PROFILE;
const relocatedTarget = { ...target, path: "/tmp/relocated-app" };
const parameters = { address: "0x1000" };
const locations = [{ kind: "address" as const, address: "0x1000" }];
cache.record({
target,
profile,
operation: "procedure_info",
parameters,
execution: createAnalysisExecution(
{ name: "main" },
ANALYSIS_SNAPSHOT_PROVIDER,
{
subject: target,
rawResult: { provider: "raw" },
locations,
analysisProfile: profile,
},
),
});
const replayedTarget = cache.lookup(
relocatedTarget,
profile,
"procedure_info",
parameters,
);
expect(replayedTarget?.subject?.path).toBe(relocatedTarget.path);
expect(replayedTarget?.rawResult).toEqual({ provider: "raw" });
expect(replayedTarget?.locations).toEqual(locations);
const childSubject = {
path: "/tmp/child.zip",
sha256: "b".repeat(64),
format: "zip" as const,
};
const childParameters = { address: "0x2000" };
cache.record({
target,
profile,
operation: "procedure_info",
parameters: childParameters,
execution: createAnalysisExecution(
{ name: "child" },
ANALYSIS_SNAPSHOT_PROVIDER,
{ subject: childSubject, analysisProfile: profile },
),
});
expect(
cache.lookup(relocatedTarget, profile, "procedure_info", childParameters)
?.subject,
).toMatchObject(childSubject);
});
describe("analysis snapshot cache capacity", () => {
it("shares capacity across query kinds while allowing replacements and fresh partitions", () => {
const cache = new AnalysisSnapshotCache();
+17 -16
View File
@@ -8,6 +8,7 @@ import {
} from "../../domain/evidenceErrors.js";
import { NoBinaryOpenError } from "../../domain/configurationErrors.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { rebindEvidenceSubjectTargetPath } from "../../domain/evidence.js";
import { err, ok, type Result } from "../../domain/result.js";
import {
analysisQueryId,
@@ -277,7 +278,21 @@ export class AnalysisSnapshotCache {
);
const cached = this.#entries.get(queryId);
if (cached === undefined) return undefined;
const subject = cached.execution.subject;
const cachedSubject = cached.execution.subject;
const subject =
cachedSubject === null
? null
: rebindEvidenceSubjectTargetPath(
{
path: cachedSubject.path,
sha256: cachedSubject.sha256,
format: cachedSubject.format,
...(cachedSubject.architecture === null
? {}
: { architecture: cachedSubject.architecture }),
},
target,
);
return structuredClone({
result: cached.execution.result,
rawResult: cached.execution.raw_result,
@@ -288,21 +303,7 @@ export class AnalysisSnapshotCache {
"Loaded from a local REA analysis snapshot; this call did not re-run the provider.",
],
locations: cached.execution.locations,
subject:
subject === null
? null
: subject.architecture === null
? {
path: subject.path,
sha256: subject.sha256,
format: subject.format,
}
: {
path: subject.path,
sha256: subject.sha256,
format: subject.format,
architecture: subject.architecture,
},
subject,
});
}
+62 -6
View File
@@ -98,6 +98,30 @@ export class BinarySession
this.#providerRouter = providerRouter;
}
/**
* Fast-path liveness gate (#1719): short-circuit only when the retained
* client is genuinely alive. The probe reads the child's operation health
* rather than the composite's unconditionally-ok `health`, so a dead child
* falls through and a real client is started; an aborted caller leaves the
* session untouched. Returns the fast-path result, or undefined to fall
* through to the slow path.
*/
async #tryRetainedFastPath(
retained: SessionBinding,
target: BinaryTarget,
options: Pick<BinarySessionOpenOptions, "signal" | "snapshot">,
): Promise<Result<BinaryTarget, AnalysisError> | undefined> {
const live = await this.#retainedClientLive(retained.client, options);
if (isAborted(options.signal))
return err(new AnalysisCancelledError("open_binary"));
if (!live) return undefined;
if (options.snapshot !== undefined) {
const imported = this.importAnalysisSnapshot(options.snapshot);
if (!imported.ok) return imported;
}
return ok(target);
}
/** Identify the provider producing evidence for this session. */
providerIdentity(operation?: AnalysisOperation): ProviderIdentity {
const route = this.#currentRoute();
@@ -247,12 +271,14 @@ export class BinarySession
if (!pendingClosed.ok) return pendingClosed;
if (isAborted(options.signal))
return err(new AnalysisCancelledError("open_binary"));
if (sameTarget && sameProfile) {
if (options.snapshot !== undefined) {
const imported = this.importAnalysisSnapshot(options.snapshot);
if (!imported.ok) return imported;
}
return ok(target);
const retained = sameTarget && sameProfile ? this.#active : undefined;
if (retained !== undefined) {
const fastPath = await this.#tryRetainedFastPath(
retained,
target,
options,
);
if (fastPath !== undefined) return fastPath;
}
const previous = this.#active;
this.#active = undefined;
@@ -703,6 +729,36 @@ export class BinarySession
);
}
/**
* Probe whether a retained provider client is still live.
*
* On production routes the client is a CompositeProvider whose `health`
* answers unconditionally ok without touching a child, so `health` alone
* cannot see a dead child. `operationHealthSnapshot` reaches the
* instantiated children instead. Only the known-live states allow the
* fast path: `exited`/`unreachable` are dead, `unknown` is never
* evidence of health (it is what a recorded but uncharacterized failure
* reports), and `not_started` is a startup that never completed — in the
* Hopper timeout case the client holds retained resources that block the
* next `start()` until `close()` retries cleanup, so treating it as live
* would skip the retire path and hand the wedged client to subsequent
* analysis. The gate fails closed: without a snapshot signal, fall back
* to the `health` admission probe.
*/
async #retainedClientLive(
client: AnalysisClient,
options: Pick<BinarySessionOpenOptions, "signal">,
): Promise<boolean> {
const snapshot = client.operationHealthSnapshot?.();
if (snapshot !== undefined)
return snapshot.state === "idle" || snapshot.state === "busy";
try {
return (await client.execute("health", {}, options)).ok;
} catch {
return false;
}
}
async #restore(
previous: SessionBinding | undefined,
): Promise<Result<null, AnalysisError>> {
@@ -0,0 +1,338 @@
import { describe, expect, it } from "vitest";
import {
createAnalysisExecution,
type AnalysisClient,
type AnalysisOperation,
type AnalysisProvider,
type AnalysisProviderCandidate,
type CapabilityDescriptor,
type ProviderAvailability,
type ProviderIdentity,
type ProviderTargetSupport,
} from "../AnalysisProvider.js";
import { createAnalysisProfile } from "../../domain/analysisProfile.js";
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
import type { ProviderOperationHealth } from "../../domain/providerOperationHealth.js";
import { err, ok } from "../../domain/result.js";
import { InvestigationRecords } from "../investigation/InvestigationRecords.js";
import { AnalysisProviderRegistry } from "./AnalysisProviderRegistry.js";
import { BinarySession } from "./BinarySession.js";
import { SessionProviderRouter } from "./SessionProviderRouter.js";
/**
* Regression coverage for https://github.com/morluto/rea/issues/1719:
* `BinarySession#open` must not short-circuit on the `sameTarget &&
* sameProfile` fast path when the retained provider client is dead. It must
* start a fresh client instead.
*
* The fixtures are production-composed: a deep provider plus an auxiliary
* provider, so `SessionProviderRouter` wraps them in a real
* `CompositeProvider` exactly like `src/composition/binary.ts` does for the
* CLI/MCP runtime. The composite's `health` answers unconditionally ok
* without touching a child, so a probe that only calls `health` cannot see a
* dead child — the liveness signal must reach the actual child through
* `operationHealthSnapshot`, the way `HopperClient.operationHealth` reports a
* recorded process exit.
*/
describe("BinarySession open liveness", () => {
it("reuses the retained client while the child stays live", async () => {
const { session, deep } = openSession();
expect((await session.open(process.execPath)).ok).toBe(true);
// Instantiate the deep child inside the composite, like a real query would.
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
const second = await session.open(process.execPath);
expect(second.ok).toBe(true);
// Still the same composite and child: nothing was retired or recreated.
expect(deep.created).toHaveLength(1);
expect(deep.created[0]?.closeCalled).toBe(false);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
});
it("starts a fresh client when the retained child died", async () => {
const { session, deep } = openSession();
expect((await session.open(process.execPath)).ok).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
// Kill the provider process out from under the session, then re-open
// the same target: the session must not short-circuit on the dead
// child — it must retire it and start a real one.
const dead = deep.created[0];
dead?.kill();
const second = await session.open(process.execPath);
expect(second.ok).toBe(true);
// The dead binding was retired, not retained.
expect(dead?.closeCalled).toBe(true);
// The next query runs against a freshly created child, not the dead one.
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(2);
});
it("treats an uncharacterized operation failure as dead", async () => {
// A recorded failure the provider could not characterize ("unknown",
// e.g. a Hopper spawn failure with no observed exit code) is never
// evidence of health: the gate fails closed.
const { session, deep } = openSession();
expect((await session.open(process.execPath)).ok).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
const dead = deep.created[0];
dead?.kill("unknown");
const second = await session.open(process.execPath);
expect(second.ok).toBe(true);
expect(dead?.closeCalled).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(2);
});
it("retires a retained client whose startup never completed", async () => {
// A Hopper startup timeout leaves the operation-health snapshot in
// "not_started" while the retained resources block the next start()
// until close() retries cleanup. Classifying that state as live would return success
// without entering the retire path, so subsequent analysis still hits
// the wedged client.
const { session, deep } = openSession();
expect((await session.open(process.execPath)).ok).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
const wedged = deep.created[0];
wedged?.kill("not_started");
const second = await session.open(process.execPath);
expect(second.ok).toBe(true);
// The wedged binding was retired (close ran, so cleanup is retried),
// not reused.
expect(wedged?.closeCalled).toBe(true);
// The next query runs against a freshly created child, not the wedged one.
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(2);
});
it("falls back to the health probe when the child reports no operation health", async () => {
// Without an operation-health hook there is no child-liveness signal,
// so the probe keeps the previous `health` admission behavior.
const { session, deep } = openSession({ operationHealth: false });
expect((await session.open(process.execPath)).ok).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
expect(deep.created).toHaveLength(1);
const second = await session.open(process.execPath);
expect(second.ok).toBe(true);
// The composite's `health` answers ok, so the live client is reused.
expect(deep.created).toHaveLength(1);
expect(deep.created[0]?.closeCalled).toBe(false);
});
it("reports cancellation without touching the session when aborted mid-probe", async () => {
const { session, deep, onProbe } = openSession();
expect((await session.open(process.execPath)).ok).toBe(true);
expect((await session.execute("address_name", {})).ok).toBe(true);
const controller = new AbortController();
// Fire the abort while the fast-path liveness probe is in flight: the
// hook runs synchronously inside the snapshot read, so the abort lands
// after the probe started but before the fast-path decision completes.
let fired = false;
onProbe(() => {
if (!fired) {
fired = true;
controller.abort();
}
});
const cancelled = await session.open(process.execPath, {
signal: controller.signal,
});
expect(cancelled).toMatchObject({
ok: false,
error: { _tag: "AnalysisCancelledError" },
});
// The dead-or-live question is moot: the session must be untouched.
expect(deep.created).toHaveLength(1);
expect(deep.created[0]?.closeCalled).toBe(false);
const reopened = await session.open(process.execPath);
expect(reopened.ok).toBe(true);
expect(deep.created).toHaveLength(1);
});
});
interface KillableChild extends AnalysisClient {
kill(state?: "exited" | "unreachable" | "unknown" | "not_started"): void;
closeCalled: boolean;
}
const deepIdentity: ProviderIdentity = {
id: "liveness-deep",
name: "Liveness deep provider",
version: "1",
};
const auxIdentity: ProviderIdentity = {
id: "liveness-aux",
name: "Liveness auxiliary provider",
version: "1",
};
const capability = (
identity: ProviderIdentity,
operation: Exclude<AnalysisOperation, "health">,
): CapabilityDescriptor => ({
provider: identity,
operation,
available: true,
reason: null,
effects: {
mutatesArtifact: false,
launchesProcess: false,
mayShowUi: false,
mayAccessNetwork: false,
mayWriteFilesystem: false,
changesPermissions: false,
requiresRoot: false,
},
limitations: [],
});
const openSession = (
options: { operationHealth?: boolean } = {},
): {
session: BinarySession;
deep: { created: KillableChild[] };
onProbe: (hook: () => void) => void;
} => {
const { provider, created, onProbe } = killableDeepProvider(options);
// Mirror the production composition: a deep provider plus auxiliary
// providers, so the route wraps them in a real CompositeProvider (whose
// trivially-ok `health` must not mask a dead child).
const aux: AnalysisProvider = {
identity: () => auxIdentity,
capabilities: () => [],
createClient: (): AnalysisClient => ({
execute: (operation) =>
Promise.resolve(
ok(createAnalysisExecution(`aux:${operation}`, auxIdentity)),
),
close: () => Promise.resolve(ok(null)),
}),
};
const session = new BinarySession(
SessionProviderRouter.selectable(new AnalysisProviderRegistry([provider]), [
aux,
]),
new InvestigationRecords(),
);
return { session, deep: { created }, onProbe };
};
const killableDeepProvider = (options: {
operationHealth?: boolean;
}): {
provider: AnalysisProviderCandidate;
created: KillableChild[];
onProbe: (hook: () => void) => void;
} => {
const created: KillableChild[] = [];
let probeHook: (() => void) | undefined;
const withOperationHealth = options.operationHealth !== false;
const provider: AnalysisProviderCandidate = {
identity: () => deepIdentity,
capabilities: () => [capability(deepIdentity, "address_name")],
inspectAvailability: (): ProviderAvailability => ({
status: "available",
code: null,
reason: null,
diagnostics: {},
}),
inspectTargetSupport: (): ProviderTargetSupport => ({
status: "supported",
code: null,
reason: null,
diagnostics: {},
}),
resolveAnalysisProfile: () =>
Promise.resolve(
ok({
profile: createAnalysisProfile(
{ id: deepIdentity.id, name: deepIdentity.name, version: "1" },
{ fixture: "liveness" },
),
}),
),
createClient: (): KillableChild => {
let alive = true;
let closeCalled = false;
let deadState: "exited" | "unreachable" | "unknown" | "not_started" =
"exited";
const client: KillableChild = {
execute: (operation) => {
return alive
? Promise.resolve(
ok(
createAnalysisExecution(
operation === "health" ? null : `liveness:${operation}`,
deepIdentity,
),
),
)
: Promise.resolve(
err(
new ProviderAdapterError(deepIdentity.id, operation, {
diagnostics: { reason: "process_killed" },
}),
),
);
},
// Genuine child-liveness signal, like HopperClient.operationHealth:
// a recorded process exit stays visible after the kill.
...(withOperationHealth
? {
operationHealthSnapshot: (): ProviderOperationHealth => {
probeHook?.();
return {
state: alive ? "idle" : deadState,
stage: alive ? null : "connection",
retryAction: alive ? null : "restart_provider",
exitCode: alive ? null : 1,
requests: [],
};
},
}
: {}),
close: () => {
closeCalled = true;
return Promise.resolve(ok(null));
},
get closeCalled() {
return closeCalled;
},
kill: (
state:
| "exited"
| "unreachable"
| "unknown"
| "not_started" = "exited",
) => {
alive = false;
deadState = state;
},
};
created.push(client);
return client;
},
};
return {
provider,
created,
onProbe: (hook: () => void) => {
probeHook = hook;
},
};
};
@@ -91,4 +91,7 @@ const parameters = (
): EvidenceObservation["parameters"] => ({
format: input.format,
integrity_policy: input.integrity_policy,
...(input.source_paths === undefined
? {}
: { source_paths: input.source_paths }),
});
@@ -4,6 +4,7 @@ import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import {
analyzeJavaScriptApplicationInputSchema,
type JavaScriptApplicationAnalysisResult,
parseOwnedJavaScriptApplicationAnalysisSteps,
} from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisOutputError } from "../../domain/analysisErrorCore.js";
@@ -20,7 +21,10 @@ import {
checkpointJavaScriptAnalysis,
completeJavaScriptAnalysisSteps,
} from "./JavaScriptAnalysisControl.js";
import { reconstructJavaScriptArtifact } from "./JavaScriptArtifactReconstruction.js";
import {
reconstructJavaScriptArtifact,
type JavaScriptArtifactReconstructionResult,
} from "./JavaScriptArtifactReconstruction.js";
import { JAVASCRIPT_APPLICATION_PROVIDER } from "../InvestigationProviders.js";
const OPERATION = "analyze_javascript_application" as const;
@@ -45,6 +49,9 @@ export const analyzeJavaScriptApplicationValidated = async (
scope: ArtifactResourceScope,
options: ExecutionOptions = {},
): Promise<Result<Evidence, AnalysisError>> => {
let reconstructed: JavaScriptArtifactReconstructionResult | undefined;
let result: JavaScriptApplicationAnalysisResult | undefined;
let evidence: Evidence | undefined;
try {
assertJavaScriptAnalysisActive(options.signal);
await options.progress?.report({
@@ -54,11 +61,14 @@ export const analyzeJavaScriptApplicationValidated = async (
message: "Inventorying and parsing application artifacts",
});
await checkpointJavaScriptAnalysis(options.signal);
const reconstructed = await reconstructJavaScriptArtifact(
reconstructed = await reconstructJavaScriptArtifact(
{
input_path: input.input_path,
format: input.format,
integrity_policy: input.integrity_policy,
...(input.source_paths === undefined
? {}
: { source_paths: input.source_paths }),
...(input.max_heap_mb === undefined
? {}
: { max_heap_mb: input.max_heap_mb }),
@@ -78,7 +88,7 @@ export const analyzeJavaScriptApplicationValidated = async (
message: "Validating result metadata and cross-graph bindings",
});
await checkpointJavaScriptAnalysis(options.signal);
const result = await completeJavaScriptAnalysisSteps(
result = await completeJavaScriptAnalysisSteps(
parseOwnedJavaScriptApplicationAnalysisSteps({
...application,
summary,
@@ -93,13 +103,12 @@ export const analyzeJavaScriptApplicationValidated = async (
message: "Creating and hashing application analysis Evidence",
});
await checkpointJavaScriptAnalysis(options.signal);
const evidence =
await createOwnedJavaScriptApplicationEvidenceCooperatively(
input,
result,
options.signal,
options.progress,
);
evidence = await createOwnedJavaScriptApplicationEvidenceCooperatively(
input,
result,
options.signal,
options.progress,
);
await checkpointJavaScriptAnalysis(options.signal);
await options.progress?.report({
phase: "analyze_javascript_application",
@@ -111,6 +120,38 @@ export const analyzeJavaScriptApplicationValidated = async (
assertJavaScriptAnalysisActive(options.signal);
return ok(evidence);
} catch (cause: unknown) {
if (
cause instanceof ArtifactReaderFailure &&
cause.reason === "cancelled" &&
reconstructed !== undefined
) {
try {
if (result === undefined) {
const { electron_summary: summary, ...application } = reconstructed;
result = await completeJavaScriptAnalysisSteps(
parseOwnedJavaScriptApplicationAnalysisSteps({
...application,
summary,
limitations: reconstructed.graph.limitations,
}),
);
}
evidence ??=
await createOwnedJavaScriptApplicationEvidenceCooperatively(
input,
result,
);
return err(
artifactFailureToOperationError(cause).retainPartialObservation(
evidence,
),
);
} catch (completionCause: unknown) {
// A validation or Evidence failure while finishing the partial result
// remains the primary failure instead of being hidden by cancellation.
cause = completionCause;
}
}
// Preserve execution constraints and their completed partial Evidence.
if (cause instanceof AnalysisError) return err(cause);
if (cause instanceof ArtifactReaderFailure)
@@ -11,7 +11,10 @@ import type { JavaScriptSemanticIr } from "../../domain/javascript/javascriptSem
import { projectJavaScriptModuleSemantics } from "../../domain/javascript/javascriptModuleSemanticIr.js";
import { parseJavaScriptSource } from "../../domain/javascript/javascriptSourceParser.js";
import { hasValidSourceMapContents } from "../../domain/sourceMapContents.js";
import { flattenSourceMapLeaves } from "../../domain/sourceMapEnvelope.js";
import {
flattenSourceMapLeaves,
sourceMapJsonText,
} from "../../domain/sourceMapEnvelope.js";
import type {
JavaScriptSourceRange,
JavaScriptSourcePoint,
@@ -137,12 +140,31 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
}> => {
const state = emptyArtifactAnalysis<JavaScriptModuleSemanticIr>();
let interruption: AnalysisError | null = null;
for (const [index, file] of fileSet.files.entries()) {
const selectedFileCount =
fileSet.files.length - (fileSet.source_selection?.unselected_files ?? 0);
let selectedFileIndex = 0;
// Caller-excluded sources are never analyzed, so neither cancellation nor
// another interruption describes them.
const recordOutsideSelection = (file: JavaScriptArtifactFile): boolean => {
if (file.text.included || file.text.reason !== "outside-analysis-scope")
return false;
state.files.push({ file, javascript: null, semantic: null });
if (file.kind === "javascript")
projection.recordUnavailableFile(
file,
"Source was not analyzed because it is outside the caller-selected source_paths.",
null,
"incomplete-module",
);
return true;
};
for (const file of fileSet.files) {
try {
await setImmediate();
await beforeFile(file, index, fileSet.files.length);
if (signal?.aborted === true)
throw new AnalysisCancelledError("analyze_javascript_application");
if (recordOutsideSelection(file)) continue;
await beforeFile(file, selectedFileIndex++, selectedFileCount);
if (file.kind !== "javascript" || !file.text.included) {
const steps = analyzeArtifactFileSteps(file, {
state,
@@ -232,7 +254,7 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
: cause instanceof AnalysisError
? cause
: null;
if (interruption !== null) {
if (interruption !== null && !recordOutsideSelection(file)) {
state.files.push({
file,
javascript: null,
@@ -266,7 +288,8 @@ export const analyzeAndAdoptJavaScriptArtifactFiles = async (
: null,
};
for (const file of fileSet.files) {
if (analyzedPaths.has(file.path)) continue;
if (analyzedPaths.has(file.path) || recordOutsideSelection(file))
continue;
state.files.push({
file,
javascript: null,
@@ -336,6 +359,10 @@ function* analyzeArtifactFileSteps<
context: ArtifactAnalysisContext<SemanticIr>,
): Generator<void, void> {
const { state } = context;
if (!file.text.included && file.text.reason === "outside-analysis-scope") {
state.files.push({ file, javascript: null, semantic: null });
return;
}
addStructuredObservations(file, state);
if (file.kind === "html" && file.text.included)
state.htmlScripts.push(...parseHtmlScripts(file.path, file.text.value));
@@ -486,12 +513,12 @@ const parseSourceMap = (
if (!file.text.included) return unavailableSourceMap(file);
let value: unknown;
try {
value = JSON.parse(file.text.value);
value = JSON.parse(sourceMapJsonText(file.text.value));
} catch (cause: unknown) {
void cause;
return invalidSourceMap(file, "Source map is not valid JSON.");
}
const maps = flattenSourceMaps(value);
const maps = flattenSourceMapLeaves(value);
if (maps === undefined)
return invalidSourceMap(file, "Source map is not a version 3 map.");
return collectSourceMapOriginals(file, maps);
@@ -556,13 +583,6 @@ const unavailableSourceMap = (
};
};
const flattenSourceMaps = (
root: unknown,
): Readonly<Record<string, unknown>>[] | undefined => {
const leaves = flattenSourceMapLeaves(root);
return leaves === undefined ? undefined : [...leaves];
};
const invalidSourceMap = (
file: JavaScriptArtifactFile,
limitation: string,
@@ -168,6 +168,7 @@ const graphCoverage = (context: JavaScriptArtifactGraphContext) => {
javascript !== null && javascript.parse_status === "partial",
);
const unknownGap =
(context.fileSet.source_selection?.unselected_files ?? 0) > 0 ||
context.analysis.files.some(
({ analysis_failure, application_projection_failure }) =>
analysis_failure !== undefined ||
@@ -244,6 +245,11 @@ const graphLimitations = (
);
return [
...context.analysis.limitations,
...(context.fileSet.source_selection === undefined
? []
: [
`Caller-selected source analysis leaves ${String(context.fileSet.source_selection.unselected_files)} inventoried files outside analysis scope. Inventory identity covers the complete artifact; relationships through unselected sources remain unknown.`,
]),
...context.snapshot.integrity_contradictions.map(
({ logical_path: path }) =>
`Artifact integrity metadata contradicts observed bytes at ${path}; the observed bytes are untrusted.`,
@@ -199,7 +199,15 @@ export const addJavaScriptArtifactFiles = (
}),
});
}
if (analyzed.analysis_failure !== undefined)
if (!file.text.included && file.text.reason === "outside-analysis-scope")
addUnavailableStaticParseScope(context, {
file,
asset: target,
operation: "select-javascript-sources",
limitation:
"Source is outside the caller-selected source_paths; it was inventoried but not analyzed.",
});
else if (analyzed.analysis_failure !== undefined)
addUnavailableStaticParseScope(context, {
file,
asset: target,
@@ -1464,6 +1464,159 @@ describe("contextual JavaScript module identity", () => {
});
});
describe("TypeScript NodeNext emitted-extension resolution", () => {
const container = "container";
it.each([
["./dep.js", "src/dep.ts"],
["./ui/Button.js", "src/ui/Button.tsx"],
["./cfg.mjs", "src/cfg.mts"],
["./legacy.cjs", "src/legacy.cts"],
] as const)(
"resolves %s to its TypeScript source counterpart",
(specifier, target) => {
const files = fileMap([
file("src/main.ts", container),
file(target, container, "export const value = 1;"),
]);
expect(
resolve({
declaredPath: specifier,
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ resolution_status: "resolved", resolved_path: target });
},
);
it.each([
["./dep.js", "src/dep.d.ts"],
["./cfg.mjs", "src/cfg.d.mts"],
["./legacy.cjs", "src/legacy.d.cts"],
] as const)(
"resolves %s to a declaration counterpart",
(specifier, target) => {
const files = fileMap([
file("src/main.ts", container),
file(target, container, "export const value = 1;"),
]);
expect(
resolve({
declaredPath: specifier,
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ resolution_status: "resolved", resolved_path: target });
},
);
it("prefers a present runtime file and keeps the fallback off for JS callers", () => {
const files = fileMap([
file("src/main.ts", container),
file("src/dep.js", container),
file("src/dep.ts", container),
file("src/js-main.js", container),
]);
expect(
resolve({
declaredPath: "./dep.js",
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({
resolution_status: "resolved",
resolved_path: "src/dep.js",
});
expect(
resolve({
declaredPath: "./dep.js",
sourcePath: "src/js-main.js",
context: "module-specifier",
moduleKind: "import",
files: fileMap([
file("src/js-main.js", container),
file("src/dep.ts", container),
]),
}),
).toMatchObject({ resolution_status: "not-found", resolved_path: null });
});
it("does not apply the mapping to URL resources or package export targets", () => {
const urlFiles = fileMap([
file("src/main.ts", container),
file("src/asset.ts", container),
]);
expect(
resolve({
declaredPath: "asset.js",
sourcePath: "src/main.ts",
context: "url-reference",
files: urlFiles,
}),
).toMatchObject({ resolution_status: "not-found", resolved_path: null });
const packageFiles = fileMap([
file("src/main.ts", container),
file(
"src/node_modules/fixture/package.json",
container,
'{"exports":"./dep.js"}',
),
file("src/node_modules/fixture/dep.ts", container),
]);
expect(
resolve({
declaredPath: "fixture",
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files: packageFiles,
}),
).toMatchObject({ resolution_status: "not-found", resolved_path: null });
});
it("does not substitute source counterparts for directory or index resolution", () => {
const files = fileMap([
file("src/main.ts", container),
file("src/lib/index.ts", container),
]);
expect(
resolve({
declaredPath: "./lib.js",
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({ resolution_status: "not-found", resolved_path: null });
});
it("resolves an emitted dotfile name within its own directory", () => {
const files = fileMap([
file("src/main.ts", container),
file("src/.ts", container),
]);
expect(
resolve({
declaredPath: "./.js",
sourcePath: "src/main.ts",
context: "module-specifier",
moduleKind: "import",
files,
}),
).toMatchObject({
resolution_status: "resolved",
resolved_path: "src/.ts",
});
});
});
describe("url-reference JavaScript artifact path resolution", () => {
it.each([
"app%00.js.map",
@@ -19,6 +19,7 @@ import {
classifyPackageImportTargetString,
lookupExactPackageImport,
} from "../../domain/javascript/javascriptPackageImportAliases.js";
import { typescriptEmittedSourceCandidates } from "../TypeScriptEmittedSourceCandidates.js";
type ArtifactPathResolutionContext =
| "package-entrypoint"
@@ -59,6 +60,7 @@ export interface ResolveArtifactPathInput {
readonly context: ArtifactPathResolutionContext;
readonly files: ReadonlyMap<string, JavaScriptArtifactFile>;
readonly htmlBaseHref?: string | null;
/** Import/require semantics; runtime resource paths omit this provenance. */
readonly moduleKind?: "import" | "require";
}
@@ -73,6 +75,21 @@ const EXTENSIONS = [
".node",
];
// A private URL namespace for logical artifact paths, including root packages.
// Encode inventory segments so filename punctuation remains literal in the base.
const PACKAGE_ARTIFACT_ROOT_URL = new URL("file:///rea-artifact/");
const packageMetadataUrl = (path: string): URL =>
new URL(
path.split("/").map(encodeURIComponent).join("/"),
PACKAGE_ARTIFACT_ROOT_URL,
);
interface PackageFileTarget {
readonly value: string;
/** Normalized URL retained until loader-specific validation and file lookup. */
readonly url: URL;
}
type CandidateResolution =
| {
readonly resolvedPath: string;
@@ -128,7 +145,16 @@ const resolvePath = (input: ResolveArtifactPathInput): CandidateResolution => {
`The exact HTML resource ${confined} was not found among the selected application files.`,
])
);
return resolveCandidate(input, confined, new Set());
return resolveCandidate(
input,
confined,
new Set(),
input.context === "module-specifier" &&
input.moduleKind !== undefined &&
(input.declaredPath.startsWith(".") || input.declaredPath.startsWith("/"))
? typescriptEmittedSourceCandidates(input.sourcePath, confined)
: [],
);
};
const rejectDeclaration = (declared: string): CandidateResolution | null => {
@@ -150,9 +176,7 @@ const contextualCandidate = (
): string | CandidateResolution => {
const { context } = input;
let declared =
context === "html-reference" ||
context === "url-reference" ||
(context === "module-specifier" && input.moduleKind !== "require")
context === "html-reference" || context === "url-reference"
? stripQueryAndFragment(input.declaredPath)
: input.declaredPath;
if (context === "html-reference") return htmlCandidate(input);
@@ -164,9 +188,9 @@ const contextualCandidate = (
"URL and Node builtin schemes are outside static artifact module resolution.",
]);
if (!declared.startsWith(".") && !declared.startsWith("/"))
return bareModuleCandidate(input, declared);
return bareModuleCandidate(input);
if (input.moduleKind !== "require") {
const decoded = decodeModuleUrlPath(declared);
const decoded = decodeModuleUrlPath(stripQueryAndFragment(declared));
if (typeof decoded !== "string") return decoded;
declared = decoded;
}
@@ -211,34 +235,51 @@ const relativeCandidate = (sourcePath: string, declared: string): string =>
const bareModuleCandidate = (
input: ResolveArtifactPathInput,
declared: string,
): CandidateResolution => {
const packageName = barePackageName(declared);
if (packageName === null || isBuiltin(declared))
const request = parseBarePackageRequest(input.declaredPath, input.moduleKind);
if (request === null || isBuiltin(input.declaredPath))
return unresolvedCandidate("external", [
"The bare specifier is a Node builtin or invalid package name.",
]);
if (input.moduleKind === "import" && packageName.includes("%"))
if (input.moduleKind === "import" && request.packageName.includes("%"))
return unresolvedCandidate("rejected", [
"Percent signs are not admitted in ES module package names.",
]);
const self = commonJsSelfCandidate(input, declared);
const self = packageSelfCandidate(input, request);
if (self !== null) return self;
const subpath =
declared === packageName ? "." : `.${declared.slice(packageName.length)}`;
// ESM constructs a package.json URL; its parsed identity is not a literal
// dependency directory. Self resolution above still uses the exact name.
let dependencyDirectory = posix.join("node_modules", request.packageName);
if (input.moduleKind === "import") {
const metadataUrl = new URL(
`./node_modules/${request.packageName}/package.json`,
PACKAGE_ARTIFACT_ROOT_URL,
);
if (!metadataUrl.pathname.endsWith("/package.json"))
return unresolvedCandidate("external", [
`ESM package name ${JSON.stringify(request.packageName)} produces a dependency URL whose pathname does not name package.json; no literal or suffix-stripped package directory was searched.`,
]);
const decoded = decodeModuleUrlPath(
metadataUrl.pathname.slice(PACKAGE_ARTIFACT_ROOT_URL.pathname.length),
);
if (typeof decoded !== "string") return decoded;
const confined = confineCandidate(decoded);
if (typeof confined !== "string") return confined;
dependencyDirectory = posix.dirname(confined);
}
const source = input.files.get(input.sourcePath);
let directory = posix.dirname(input.sourcePath);
while (true) {
const candidate = posix.join(directory, "node_modules", packageName);
const candidate = posix.join(directory, dependencyDirectory);
const metadata = readPackageMetadata(input, candidate);
if (packageExportsEligible(input) && metadata.kind !== "absent")
return resolvePackageSpecifier(input, candidate, subpath, metadata);
if (subpath === ".") {
if (request.dependencyExportsEligible && metadata.kind !== "absent")
return resolvePackageDependency(input, candidate, request, metadata);
if (request.packageSubpath === ".") {
const direct = resolveFileCandidates(input, fileCandidates(candidate));
if (direct !== null) return direct;
}
if (hasContainerDirectory(input.files, candidate, source?.container_sha256))
return resolvePackageSpecifier(input, candidate, subpath, metadata);
return resolvePackageDependency(input, candidate, request, metadata);
if (directory === "." || directory === "") break;
directory = posix.dirname(directory);
}
@@ -304,20 +345,25 @@ const resolvePackageImportAlias = (
return unresolvedCandidate("external", [
`Wildcard package import aliases remain unresolved; package metadata ${metadata.path} has no exact imports key ${specifier}.`,
]);
const selected = selectPackageImportTarget(lookup.target, input.moduleKind);
if (selected.status === "rejected")
const selected = selectExportTarget(
lookup.target,
packageExportConditions(input.moduleKind),
packageMetadataUrl(metadata.path),
"imports",
);
if (selected.kind === "invalid" || selected.kind === "invalid-config")
return unresolvedCandidate("rejected", [
`Package metadata ${metadata.path}: ${selected.limitation}`,
]);
if (selected.status === "unmatched")
if (selected.kind === "unmatched")
return unresolvedCandidate("external", [
`Package metadata ${metadata.path} declares no imports target for alias ${specifier} under the active conditions; it declares ${selected.declared.join(", ") || "no conditions"}.`,
]);
if (selected.status === "blocked")
if (selected.kind === "blocked")
return unresolvedCandidate("external", [
`Package metadata ${metadata.path} blocks import alias ${specifier} with ${selected.cause === "null" ? "an explicit null imports target" : "an empty imports target array"}.`,
]);
if (selected.status === "non-relative") {
if (selected.kind === "non-relative") {
const target = classifyPackageImportTargetString(selected.value);
return unresolvedCandidate("external", [
target.kind === "recursive"
@@ -325,74 +371,47 @@ const resolvePackageImportAlias = (
: `External-package import targets remain unresolved; package metadata ${metadata.path} maps ${specifier} to ${JSON.stringify(target.target)}.`,
]);
}
return resolveExportTarget(input, metadata.path, selected.value, "imports");
};
const selectPackageImportTarget = (
value: unknown,
moduleKind: ResolveArtifactPathInput["moduleKind"],
):
| { readonly status: "value"; readonly value: string }
| { readonly status: "non-relative"; readonly value: string }
| { readonly status: "blocked"; readonly cause: PackageTargetBlockCause }
| { readonly status: "unmatched"; readonly declared: readonly string[] }
| { readonly status: "rejected"; readonly limitation: string } => {
const selected = selectExportTarget(
value,
packageExportConditions(moduleKind),
"imports",
);
if (selected.kind === "invalid" || selected.kind === "invalid-config")
return { status: "rejected", limitation: selected.limitation };
if (selected.kind === "unmatched")
return {
status: "unmatched",
declared: selected.declared,
};
if (selected.kind === "blocked")
return { status: "blocked", cause: selected.cause };
if (selected.kind === "non-relative")
return { status: "non-relative", value: selected.value };
return { status: "value", value: selected.value };
return resolveExportTarget(input, metadata.path, selected, "imports");
};
/** Self references use the nearest package scope before dependency lookup. */
const commonJsSelfCandidate = (
const packageSelfCandidate = (
input: ResolveArtifactPathInput,
declared: string,
request: BarePackageRequest,
): CandidateResolution | null => {
if (input.moduleKind !== "require" || !input.files.has(input.sourcePath))
if (input.moduleKind === undefined || !input.files.has(input.sourcePath))
return null;
const metadata = enclosingPackageMetadata(input);
if (metadata.kind === "none") return null;
if (metadata.kind === "unavailable")
return unresolvedCandidate("unavailable", [metadata.limitation]);
if (
metadata.name === undefined ||
metadata.exports === undefined ||
(declared !== metadata.name && !declared.startsWith(`${metadata.name}/`))
)
if (metadata.name === undefined || metadata.exports === undefined)
return null;
const subpath =
declared === metadata.name
? "."
: `.${declared.slice(metadata.name.length)}`;
let subpath: string;
if (input.moduleKind === "import") {
if (metadata.name !== request.packageName) return null;
subpath = request.packageSubpath;
} else if (input.declaredPath === metadata.name) {
subpath = ".";
} else if (input.declaredPath.startsWith(`${metadata.name}/`)) {
subpath = `.${input.declaredPath.slice(metadata.name.length)}`;
} else return null;
// Self resolution does not apply the CommonJS dependency-name matcher.
return resolvePackageExports(input, metadata.path, subpath, metadata.exports);
};
/** Package roots and exact subpaths share metadata and exports selection. */
const resolvePackageSpecifier = (
/** Resolve an installed dependency using one parsed request and its metadata. */
const resolvePackageDependency = (
input: ResolveArtifactPathInput,
directory: string,
subpath: string,
request: BarePackageRequest,
metadata: PackageMetadata,
): CandidateResolution => {
const exportsEligible = packageExportsEligible(input);
if (exportsEligible && metadata.kind === "unavailable")
const subpath = request.packageSubpath;
if (request.dependencyExportsEligible && metadata.kind === "unavailable")
return unresolvedCandidate("unavailable", [metadata.limitation]);
const rawExports =
exportsEligible && metadata.kind === "available"
request.dependencyExportsEligible && metadata.kind === "available"
? metadata.exports
: undefined;
if (rawExports === undefined) {
@@ -401,8 +420,17 @@ const resolvePackageSpecifier = (
resolveFileCandidates(input, fileCandidates(directory)) ??
resolveDirectory(input, directory, metadata, new Set())
);
// Preserve the trailing slash for a package directory request.
const candidate = confineCandidate(`${directory}/${subpath.slice(2)}`);
// Resolve legacy ESM subpaths as URLs: suffixes identify the loaded module
// but do not belong to its filesystem path. Keep the declared specifier on
// the relationship; only derive the inventory lookup path here.
const fileSubpath =
input.moduleKind === "import"
? decodeModuleUrlPath(stripQueryAndFragment(subpath.slice(2)))
: subpath.slice(2);
if (typeof fileSubpath !== "string") return fileSubpath;
const candidate = confineCandidate(`${directory}/${fileSubpath}`);
if (input.moduleKind === "import" && typeof candidate === "string")
return resolveLegacyEsmSubpath(input, candidate);
return typeof candidate === "string"
? resolveCandidate(input, candidate, new Set())
: candidate;
@@ -415,13 +443,33 @@ const resolvePackageSpecifier = (
);
};
/** Legacy ESM package subpaths are exact URLs, never extension or directory lookups. */
const resolveLegacyEsmSubpath = (
input: ResolveArtifactPathInput,
candidate: string,
): CandidateResolution => {
const file = resolveFileCandidates(input, [candidate]);
if (file !== null) return file;
const source = input.files.get(input.sourcePath);
if (hasContainerDirectory(input.files, candidate, source?.container_sha256))
return unresolvedCandidate("rejected", [
`ESM package subpath ${candidate} names a directory; directory imports are unsupported and package main or index lookup was not attempted.`,
]);
return exactPackageTargetNotFound(candidate, "ESM subpath");
};
const resolvePackageExports = (
input: ResolveArtifactPathInput,
packagePath: string,
subpath: string,
rawExports: string | object,
): CandidateResolution => {
const exported = selectPackageExport(rawExports, subpath, input.moduleKind);
const exported = selectPackageExport(
rawExports,
subpath,
input.moduleKind,
packagePath,
);
if (exported.status === "rejected")
return unresolvedCandidate("rejected", [
`Package metadata ${packagePath}: ${exported.limitation}`,
@@ -434,45 +482,78 @@ const resolvePackageExports = (
return unresolvedCandidate("external", [
`Directory package metadata ${packagePath} blocks subpath ${subpath} with ${exported.cause === "null" ? "an explicit null exports target" : "an empty exports target array"}.`,
]);
return resolveExportTarget(input, packagePath, exported.value);
return resolveExportTarget(input, packagePath, exported);
};
/** An exports or import target is a URL naming one exact file for both Node loaders. */
const resolveExportTarget = (
input: ResolveArtifactPathInput,
packagePath: string,
target: string,
target: PackageFileTarget,
mapKind: "exports" | "imports" = "exports",
): CandidateResolution => {
const rejected = rejectDeclaration(target);
if (rejected !== null) return rejected;
const decoded = decodeModuleUrlPath(stripQueryAndFragment(target));
if (typeof decoded !== "string") return decoded;
const candidate = confineCandidate(relativeCandidate(packagePath, decoded));
if (typeof candidate !== "string") return candidate;
const withTargetSource = (
outcome: CandidateResolution,
): CandidateResolution => ({
...outcome,
limitations: outcome.limitations.map(
(reason) =>
`Package metadata ${packagePath}, ${mapKind} target ${JSON.stringify(target.value)}: ${reason}`,
),
});
// Node's CommonJS finalizer checks the serialized URL, including suffixes;
// its ESM finalizer checks only the pathname. Neither error retries arrays.
if (input.moduleKind === "require" && /%(?:2f|5c)/iu.test(target.url.href))
return withTargetSource(
unresolvedCandidate("rejected", [
"The selected package URL contains encoded separator bytes.",
]),
);
const encodedPath = target.url.pathname.slice(
PACKAGE_ARTIFACT_ROOT_URL.pathname.length,
);
const rejected = rejectDeclaration(encodedPath);
if (rejected !== null) return withTargetSource(rejected);
const decoded = decodeModuleUrlPath(encodedPath);
if (typeof decoded !== "string") return withTargetSource(decoded);
const candidate = confineCandidate(decoded);
if (typeof candidate !== "string") return withTargetSource(candidate);
return (
resolveFileCandidates(input, [candidate]) ??
exactPackageTargetNotFound(candidate, mapKind)
withTargetSource(exactPackageTargetNotFound(candidate, mapKind))
);
};
const barePackageName = (specifier: string): string | null => {
const segments = specifier.split("/");
if (specifier.startsWith("@"))
return segments.length >= 2 && segments[0] !== "" && segments[1] !== ""
? `${segments[0]}/${segments[1]}`
: null;
return segments[0] === "" ? null : (segments[0] ?? null);
};
/** Derived package identity and dependency policy; input retains the raw request. */
interface BarePackageRequest {
readonly packageName: string;
readonly packageSubpath: string;
readonly dependencyExportsEligible: boolean;
}
const packageExportsEligible = (input: ResolveArtifactPathInput): boolean => {
if (input.moduleKind !== "require") return true;
const parseBarePackageRequest = (
specifier: string,
moduleKind: ResolveArtifactPathInput["moduleKind"],
): BarePackageRequest | null => {
const segments = specifier.split("/");
let packageName = segments[0] ?? "";
if (packageName === "") return null;
if (specifier.startsWith("@")) {
const packageSegment = segments[1];
if (packageSegment === undefined || packageSegment === "") return null;
packageName = `${packageName}/${packageSegment}`;
}
// Node's CommonJS exports matcher excludes percent-containing package names.
// Compare the captured name so a scoped name cannot fall back to its scope.
const match = /^((?:@[^/\\%]+\/)?[^./\\%][^/\\%]*)(\/.*)?$/.exec(
input.declaredPath,
);
return match?.[1] === barePackageName(input.declaredPath);
const dependencyExportsEligible =
moduleKind !== "require" ||
/^((?:@[^/\\%]+\/)?[^./\\%][^/\\%]*)(\/.*)?$/.exec(specifier)?.[1] ===
packageName;
return {
packageName,
packageSubpath: `.${specifier.slice(packageName.length)}`,
dependencyExportsEligible,
};
};
const hasContainerDirectory = (
@@ -669,8 +750,12 @@ const resolveCandidate = (
input: ResolveArtifactPathInput,
candidate: string,
packageChain: ReadonlySet<string>,
emittedSourceCandidates: readonly string[] = [],
): CandidateResolution =>
resolveFileCandidates(input, fileCandidates(candidate)) ??
resolveFileCandidates(input, [
...fileCandidates(candidate),
...emittedSourceCandidates,
]) ??
resolveDirectory(
input,
candidate,
@@ -763,7 +848,7 @@ const indexCandidates = (candidate: string): readonly string[] =>
type PackageTargetBlockCause = "null" | "empty-array";
type PackageExportOutcome =
| { readonly status: "value"; readonly value: string }
| ({ readonly status: "value" } & PackageFileTarget)
| { readonly status: "rejected"; readonly limitation: string }
| { readonly status: "blocked"; readonly cause: PackageTargetBlockCause }
| { readonly status: "unmatched"; readonly declared: readonly string[] };
@@ -773,6 +858,7 @@ const selectPackageExport = (
value: unknown,
subpath: string,
moduleKind: ResolveArtifactPathInput["moduleKind"],
packagePath: string,
): PackageExportOutcome => {
let target = value;
let declared: string[] = [];
@@ -799,6 +885,7 @@ const selectPackageExport = (
const selected = selectExportTarget(
target,
packageExportConditions(moduleKind),
packageMetadataUrl(packagePath),
);
if (selected.kind === "invalid" || selected.kind === "invalid-config")
return { status: "rejected", limitation: selected.limitation };
@@ -814,11 +901,15 @@ const selectPackageExport = (
status: "rejected",
limitation: `The package exports target ${JSON.stringify(selected.value)} must start with "./".`,
};
return { status: "value", value: selected.value };
return {
status: "value",
value: selected.value,
url: selected.url,
};
};
type ExportTargetOutcome =
| { readonly kind: "value"; readonly value: string }
| ({ readonly kind: "value" } & PackageFileTarget)
| { readonly kind: "non-relative"; readonly value: string }
| { readonly kind: "blocked"; readonly cause: PackageTargetBlockCause }
| { readonly kind: "unmatched"; readonly declared: readonly string[] }
@@ -853,6 +944,7 @@ const forbiddenExportSegment = (value: string): string | undefined =>
const selectStringPackageTarget = (
value: string,
packageUrl: URL,
mapKind: "exports" | "imports",
): ExportTargetOutcome => {
if (mapKind === "imports") {
@@ -867,18 +959,33 @@ const selectStringPackageTarget = (
} else if (!value.startsWith("./"))
return invalidExportTarget(value, 'must start with "./"', mapKind);
const forbidden = forbiddenExportSegment(value);
return forbidden === undefined
? { kind: "value", value }
: invalidExportTarget(
value,
`contains forbidden path segment ${JSON.stringify(forbidden)} before URL normalization`,
mapKind,
);
if (forbidden !== undefined)
return invalidExportTarget(
value,
`contains forbidden path segment ${JSON.stringify(forbidden)} before URL normalization`,
mapKind,
);
// Node constructs the URL before deciding whether this array entry is an
// invalid package target. Tabs/newlines and dot segments normalize here;
// percent decoding and existence checks belong after target selection.
const url = new URL(value, packageUrl);
if (!url.pathname.startsWith(new URL(".", packageUrl).pathname))
return invalidExportTarget(
value,
"escapes the package root after URL normalization",
mapKind,
);
return {
kind: "value",
value,
url,
};
};
const selectArrayPackageTarget = (
value: readonly unknown[],
conditions: ReadonlySet<string>,
packageUrl: URL,
mapKind: "exports" | "imports",
): ExportTargetOutcome => {
if (value.length === 0) return { kind: "blocked", cause: "empty-array" };
@@ -887,7 +994,7 @@ const selectArrayPackageTarget = (
| undefined;
const declared = new Set<string>();
for (const entry of value) {
const nested = selectExportTarget(entry, conditions, mapKind);
const nested = selectExportTarget(entry, conditions, packageUrl, mapKind);
// Node's array fallback catches invalid targets, not invalid configuration.
if (nested.kind === "invalid-config") return nested;
if (nested.kind === "unmatched") {
@@ -913,13 +1020,14 @@ const selectArrayPackageTarget = (
const selectExportTarget = (
value: unknown,
conditions: ReadonlySet<string>,
packageUrl: URL,
mapKind: "exports" | "imports" = "exports",
): ExportTargetOutcome => {
if (typeof value === "string")
return selectStringPackageTarget(value, mapKind);
return selectStringPackageTarget(value, packageUrl, mapKind);
if (value === null) return { kind: "blocked", cause: "null" };
if (Array.isArray(value))
return selectArrayPackageTarget(value, conditions, mapKind);
return selectArrayPackageTarget(value, conditions, packageUrl, mapKind);
if (typeof value !== "object")
return invalidExportTarget(
value,
@@ -938,7 +1046,7 @@ const selectExportTarget = (
for (const [condition, target] of Object.entries(value)) {
declared.add(condition);
if (!conditions.has(condition)) continue;
const nested = selectExportTarget(target, conditions, mapKind);
const nested = selectExportTarget(target, conditions, packageUrl, mapKind);
if (nested.kind !== "unmatched") return nested;
for (const name of nested.declared) declared.add(name);
}
@@ -973,12 +1081,12 @@ const notFoundCandidate = (): CandidateResolution => ({
const exactPackageTargetNotFound = (
path: string,
mapKind: "exports" | "imports",
targetKind: "exports" | "imports" | "ESM subpath",
): CandidateResolution => ({
resolvedPath: null,
status: "not-found",
limitations: [
`The selected package ${mapKind === "imports" ? "import" : "exports"} target ${path} does not exist in the inventoried artifact container; ${mapKind === "imports" ? "import" : "exports"} targets are exact files, so no extension or directory index was tried.`,
`The selected package ${targetKind === "imports" ? "import" : targetKind} target ${path} does not exist in the inventoried artifact container; ${targetKind === "imports" ? "import" : targetKind} targets are exact files, so no extension or directory index was tried.`,
],
});
@@ -1,5 +1,6 @@
import { constants } from "node:fs";
import { access, lstat, realpath } from "node:fs/promises";
import { setImmediate } from "node:timers/promises";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import {
@@ -10,9 +11,11 @@ import {
import { createJavaScriptArtifactReader as createReader } from "../../artifacts/javascript/JavaScriptArtifactReader.js";
import type { JavaScriptApplicationGraph } from "../../domain/javascript/javascriptApplicationGraph.js";
import type { JavaScriptSemanticGraph } from "../../domain/javascript/javascriptSemanticGraph.js";
import type { JavaScriptSourceSelection } from "../../domain/javascript/javascriptSourceSelection.js";
import type { ElectronBoundarySummary } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { parseOwnedJavaScriptApplicationAnalysisSteps } from "../../domain/javascript/javascriptApplicationAnalysis.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import { ArtifactOperationError } from "../../domain/artifactOperationError.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import type { ArtifactInventoryPartialObservation } from "../../domain/artifactPartialObservation.js";
import type { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
@@ -43,6 +46,7 @@ export interface JavaScriptArtifactReconstructionResult {
readonly inventory_manifest_id: string;
readonly inventory_graph_sha256: string;
readonly integrity_contradictions: ArtifactInventorySnapshot["integrity_contradictions"];
readonly source_selection?: JavaScriptSourceSelection;
readonly graph: JavaScriptApplicationGraph;
readonly semantic_graph: JavaScriptSemanticGraph;
readonly electron_summary: ElectronBoundarySummary;
@@ -68,19 +72,17 @@ export const reconstructJavaScriptArtifact = async (
progress?: ProgressReporter,
): Promise<JavaScriptArtifactReconstructionResult> =>
scope.run(async () => {
let effectiveSignal = signal;
let effectiveProgress = progress;
const reportPhase = async (
phase: string,
message: string,
): Promise<void> => {
await effectiveProgress?.report({
await progress?.report({
phase,
completed: 0,
total: 1,
message,
});
await checkpointJavaScriptAnalysis(effectiveSignal);
await checkpointJavaScriptAnalysis(signal);
};
const input = javascriptArtifactReconstructionInputSchema.parse(rawInput);
abortIfNeeded(signal);
@@ -119,6 +121,7 @@ export const reconstructJavaScriptArtifact = async (
snapshot,
scope,
signal,
input.source_paths,
);
// Release acquisition before analysis consumes these owned file facts.
const earlyCleanup = await scope.release(readerOwner);
@@ -134,7 +137,7 @@ export const reconstructJavaScriptArtifact = async (
);
await reportPhase(
"parse_javascript_sources",
`Parsing and projecting ${String(files.files.length)} application source files`,
`Parsing and projecting ${String(files.files.length - (files.source_selection?.unselected_files ?? 0))} selected application source files`,
);
const semanticProjection = createJavaScriptSemanticGraphProjection();
const worker = createJavaScriptSourceAnalysis(
@@ -182,14 +185,24 @@ export const reconstructJavaScriptArtifact = async (
}
}
const analysis = outcome.analysis;
const interruption = cleanupFailure ?? outcome.interruption;
if (interruption !== null && interruption !== undefined) {
effectiveSignal = undefined;
effectiveProgress = undefined;
}
await checkpointJavaScriptAnalysis(effectiveSignal);
abortIfNeeded(effectiveSignal);
await reportPhase(
let interruption = cleanupFailure ?? outcome.interruption;
// Source facts are owned now. Finish their projection cooperatively so a
// later cancellation can retain Evidence instead of discarding those facts.
const checkpointCompletedAnalysis = async (): Promise<void> => {
await setImmediate();
if (interruption == null && signal?.aborted === true)
interruption = new ArtifactOperationError(OPERATION, "cancelled");
};
const reportCompletedPhase = async (
phase: string,
message: string,
): Promise<void> => {
await checkpointCompletedAnalysis();
if (interruption == null)
await progress?.report({ phase, completed: 0, total: 1, message });
await checkpointCompletedAnalysis();
};
await reportCompletedPhase(
"build_javascript_application_graph",
"Constructing application and Electron boundary relationships",
);
@@ -198,15 +211,14 @@ export const reconstructJavaScriptArtifact = async (
files,
analysis,
);
await reportPhase(
await reportCompletedPhase(
"seal_javascript_application_graph",
"Sealing the validated application graph",
);
const graph = await completeJavaScriptAnalysisSteps(
applicationGraphSteps,
effectiveSignal,
);
await reportPhase(
await reportCompletedPhase(
"build_javascript_semantic_graph",
"Binding and validating static semantic relationships",
);
@@ -214,15 +226,13 @@ export const reconstructJavaScriptArtifact = async (
snapshot.manifest.root_sha256,
graph,
);
await reportPhase(
await reportCompletedPhase(
"seal_javascript_semantic_graph",
"Validating and sealing the semantic graph",
);
const semanticGraph = await completeJavaScriptAnalysisSteps(
semanticGraphSteps,
effectiveSignal,
);
await checkpointJavaScriptAnalysis(effectiveSignal);
const semanticGraph =
await completeJavaScriptAnalysisSteps(semanticGraphSteps);
await checkpointCompletedAnalysis();
const completed = {
input_path: path,
format,
@@ -230,6 +240,9 @@ export const reconstructJavaScriptArtifact = async (
inventory_manifest_id: snapshot.manifest.manifest_id,
inventory_graph_sha256: snapshot.manifest.graph_sha256,
integrity_contradictions: snapshot.integrity_contradictions,
...(files.source_selection === undefined
? {}
: { source_selection: files.source_selection }),
graph,
semantic_graph: semanticGraph,
electron_summary: summarizeElectronBoundaries(analysis),
@@ -1,12 +1,14 @@
import { z } from "zod";
import { artifactIntegrityPolicySchema } from "../../domain/artifactIntegrityPolicy.js";
import { javaScriptAnalysisResourceControlsSchema } from "../../domain/javascript/javascriptAnalysisResourceControls.js";
import { javaScriptSourcePathsSchema } from "../../domain/javascript/javascriptSourceSelection.js";
/** Local ASAR/directory reconstruction request. */
export const javascriptArtifactReconstructionInputSchema = z.strictObject({
input_path: z.string().min(1),
format: z.enum(["auto", "asar", "directory"]).default("auto"),
integrity_policy: artifactIntegrityPolicySchema,
source_paths: javaScriptSourcePathsSchema.optional(),
...javaScriptAnalysisResourceControlsSchema.shape,
});
@@ -130,6 +130,16 @@ export const buildJavaScriptSemanticGraph = ({
for (const analyzed of analysis.files) {
if (analyzed.semantic !== null)
projection.projectFile(analyzed.file, analyzed.semantic.ir);
else if (
!analyzed.file.text.included &&
analyzed.file.text.reason === "outside-analysis-scope"
)
projection.recordUnavailableFile(
analyzed.file,
"Source was not analyzed because it is outside the caller-selected source_paths.",
null,
"incomplete-module",
);
}
return projection.finish(rootArtifactSha256, applicationGraph);
};
@@ -73,11 +73,11 @@ describe("JebAnalysisService admission", () => {
const requests: JebRequest[] = [];
const service = new JebAnalysisService(fakePort(requests));
const result = await service.execute("list_jeb_units", {});
expect(result.ok).toBe(true);
if (!result.ok) return;
expect(requests[0]?.operation).toBe("list_jeb_units");
if (!result.ok) throw result.error;
expect(requests).toHaveLength(1);
const request = requests[0];
if (request?.operation !== "list_jeb_units") return;
if (request?.operation !== "list_jeb_units")
throw new Error("Expected the list_jeb_units provider request");
expect(request.input.index).toBe(0);
expect(request.input.count).toBe(100);
expect(result.value.normalized_result).toEqual({ observed: true });
+31 -3
View File
@@ -26,11 +26,35 @@ export interface ArtifactEntry {
readonly adapterKey: string;
/** Filesystem identity captured during traversal, when supplied by an adapter. */
readonly sourceIdentity?: {
readonly device: number;
readonly inode: number;
readonly device: bigint;
readonly inode: bigint;
readonly size: bigint;
readonly mode: bigint;
readonly mtimeNs: bigint;
readonly ctimeNs: bigint;
};
}
/** Compare captured filesystem identity and mutable metadata at a read boundary. */
export const sameArtifactSourceIdentity = (
expected: ArtifactEntry["sourceIdentity"],
observed: {
readonly dev: bigint;
readonly ino: bigint;
readonly size: bigint;
readonly mode: bigint;
readonly mtimeNs: bigint;
readonly ctimeNs: bigint;
},
): boolean =>
expected !== undefined &&
observed.dev === expected.device &&
observed.ino === expected.inode &&
observed.size === expected.size &&
observed.mode === expected.mode &&
observed.mtimeNs === expected.mtimeNs &&
observed.ctimeNs === expected.ctimeNs;
/** Snapshot mutable entry metadata before the producer later validates open(). */
export const copyArtifactEntry = (entry: ArtifactEntry): ArtifactEntry => ({
...entry,
@@ -60,7 +84,11 @@ export const sameArtifactEntry = (
(limitation, index) => limitation === produced.limitations[index],
) &&
entry.sourceIdentity?.device === produced.sourceIdentity?.device &&
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode;
entry.sourceIdentity?.inode === produced.sourceIdentity?.inode &&
entry.sourceIdentity?.size === produced.sourceIdentity?.size &&
entry.sourceIdentity?.mode === produced.sourceIdentity?.mode &&
entry.sourceIdentity?.mtimeNs === produced.sourceIdentity?.mtimeNs &&
entry.sourceIdentity?.ctimeNs === produced.sourceIdentity?.ctimeNs;
/** Read-only adapter over one directory, archive, or virtual container. */
export interface ArtifactReader {
+238 -110
View File
@@ -1,10 +1,4 @@
import {
constants,
createReadStream,
createWriteStream,
type ReadStream,
type Stats,
} from "node:fs";
import { constants, createWriteStream, type Stats } from "node:fs";
import {
lstat,
mkdtemp,
@@ -46,51 +40,40 @@ export class AsarArtifactReader implements ArtifactReader {
#archiveSize: number | undefined;
#headerSize: number | undefined;
readonly #entries = new Map<string, AsarEntryState>();
readonly #unpackedHandles = new Map<
UnpackedFileHandle,
{
readonly path: string;
readonly owner: OwnedFileHandle<UnpackedFileHandle>;
}
>();
readonly #ownedHandles = new Map<SharedAsarFileHandle, OwnedAsarHandle>();
#snapshot: { readonly path: string; readonly sha256: string } | undefined;
#snapshotRoot: string | undefined;
#snapshotOperation: Promise<void> = Promise.resolve();
constructor(
private readonly path: string,
private readonly openUnpackedFile: OpenAsarUnpackedFile = open,
private readonly removeSnapshot: (path: string) => Promise<void> = (path) =>
rm(path, { recursive: true, force: true }),
private readonly options: AsarArtifactReaderOptions = {},
) {}
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
let paths: readonly AsarListedPath[];
try {
await this.prepareContainer(undefined, signal);
const snapshotPath = this.#snapshotPath();
this.#resetArchiveState();
this.#entries.clear();
const headerHandle = await open(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
try {
const admitted = await admitAsarHeader(headerHandle);
paths = await this.#withSnapshotOperation(async () => {
const snapshotPath = this.#snapshotPath();
this.#resetArchiveState();
this.#entries.clear();
const admitted = await this.#admitHeader(snapshotPath);
this.#archiveSize = admitted.archiveBytes;
this.#headerSize = admitted.headerBytes;
} finally {
await headerHandle.close();
}
abortIfNeeded(signal);
paths = orderAsarListing(listPackage(snapshotPath, { isPack: false }));
const archiveMetadata = await lstat(snapshotPath);
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
throw new ArtifactReaderFailure(
"format",
"ASAR container is not a regular file",
abortIfNeeded(signal);
const ordered = orderAsarListing(
listPackage(snapshotPath, { isPack: false }),
);
this.#archiveSize = archiveMetadata.size;
const archiveMetadata = await lstat(snapshotPath);
if (!archiveMetadata.isFile() || archiveMetadata.isSymbolicLink())
throw new ArtifactReaderFailure(
"format",
"ASAR container is not a regular file",
);
this.#archiveSize = archiveMetadata.size;
return ordered;
}, signal);
} catch (cause: unknown) {
throw asarFailure(this.path, "inventory", cause);
}
@@ -142,7 +125,47 @@ export class AsarArtifactReader implements ArtifactReader {
}
}
async #admitHeader(snapshotPath: string) {
const headerHandle = await this.#openOwned(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
let admission:
| {
readonly ok: true;
readonly value: Awaited<ReturnType<typeof admitAsarHeader>>;
}
| { readonly ok: false; readonly cause: unknown };
try {
admission = { ok: true, value: await admitAsarHeader(headerHandle) };
} catch (cause: unknown) {
admission = { ok: false, cause };
}
try {
await this.#closeOwnedHandle(headerHandle);
} catch (cleanupCause: unknown) {
if (!admission.ok)
throw ArtifactReaderFailure.withCleanup(
admission.cause,
ArtifactReaderFailure.cleanupObservation(cleanupCause, snapshotPath),
);
throw cleanupCause;
}
if (!admission.ok) throw admission.cause;
return admission.value;
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
return this.#withSnapshotOperation(
() => this.#openEntry(entry, signal),
signal,
);
}
async #openEntry(
entry: ArtifactEntry,
signal?: AbortSignal,
): Promise<Readable> {
abortIfNeeded(signal);
const state = this.#entries.get(entry.adapterKey);
if (state === undefined && entry.kind !== "file")
@@ -160,10 +183,16 @@ export class AsarArtifactReader implements ArtifactReader {
"integrity",
`ASAR entry metadata changed since inventory: ${state.entry.path}`,
);
const { metadata } = state;
const producedEntry = state.entry;
if (producedEntry.unpacked)
return this.#openUnpackedStream(producedEntry, signal);
if (state.entry.unpacked)
return this.#openUnpackedStream(state.entry, signal);
return this.#openPackedStream(state, signal);
}
async #openPackedStream(
state: AsarEntryState,
signal?: AbortSignal,
): Promise<Readable> {
const { metadata, entry: producedEntry } = state;
const archiveSize = this.#archiveSize;
const headerSize = this.#headerSize;
const offset =
@@ -187,10 +216,11 @@ export class AsarArtifactReader implements ArtifactReader {
"format",
`ASAR entry range is outside its container: ${producedEntry.path}`,
);
let handle: FileHandle | undefined;
const snapshotPath = this.#snapshotPath();
let handle: SharedAsarFileHandle | undefined;
try {
const openedHandle = await open(
this.#snapshotPath(),
const openedHandle = await this.#openOwned(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
handle = openedHandle;
@@ -202,25 +232,48 @@ export class AsarArtifactReader implements ArtifactReader {
`ASAR container changed before read: ${producedEntry.path}`,
);
if (metadata.size === 0) {
await closeAsarHandle(() => openedHandle.close(), signal);
handle = undefined;
await closeAsarHandle(
() => this.#closeOwnedHandle(openedHandle),
signal,
);
return Readable.from([]);
}
const source = openedHandle.createReadStream({
const source = this.#ownedReadStream(openedHandle, {
start,
end: start + metadata.size - 1,
autoClose: true,
signal,
});
return readValidatedAsarEntry(
handle = undefined;
const output = readValidatedAsarEntry(
source,
metadata.size,
producedEntry.path,
this.path,
signal,
);
this.#setOwnedOutput(openedHandle, output);
return output;
} catch (cause: unknown) {
await handle?.close().catch(() => undefined);
throw asarFailure(this.path, `read ${producedEntry.path}`, cause);
const failure = asarFailure(
this.path,
`read ${producedEntry.path}`,
cause,
);
if (handle !== undefined) {
try {
await this.#closeOwnedHandle(handle);
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
snapshotPath,
),
);
}
}
throw failure;
}
}
@@ -246,49 +299,37 @@ export class AsarArtifactReader implements ArtifactReader {
}
/** Read the same captured container that supplies headers and packed members. */
async openContainer(signal?: AbortSignal): Promise<ReadStream> {
async openContainer(signal?: AbortSignal): Promise<Readable> {
await this.prepareContainer(undefined, signal);
return createReadStream(
this.#snapshotPath(),
signal === undefined ? {} : { signal },
);
return this.#withSnapshotOperation(async () => {
abortIfNeeded(signal);
const snapshotPath = this.#snapshotPath();
const handle = await this.#openOwned(
snapshotPath,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
return this.#ownedReadStream(handle, { signal });
}, signal);
}
async close(): Promise<void> {
await this.#withSnapshotOperation(async () => {
let cleanupFailure: ArtifactReaderFailure | undefined;
for (const [handle, { path }] of this.#unpackedHandles) {
try {
await this.#closeUnpackedHandle(handle);
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, path),
);
}
}
try {
await this.#closeSnapshot();
} catch (cause: unknown) {
cleanupFailure = ArtifactReaderFailure.withCleanup(
cleanupFailure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, this.path),
);
}
if (cleanupFailure !== undefined) throw cleanupFailure;
});
await this.#withSnapshotOperation(() => this.#closeSnapshot());
}
async #captureSnapshot(signal?: AbortSignal): Promise<void> {
let inputHandle: SharedAsarFileHandle | undefined;
try {
this.#snapshotRoot = await mkdtemp(join(tmpdir(), "rea-asar-snapshot-"));
const snapshotPath = join(this.#snapshotRoot, "container.asar");
const hash = createHash("sha256");
const handle = await open(
inputHandle = await this.#openOwned(
this.path,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const source = handle.createReadStream();
const source = this.#ownedReadStream(inputHandle, {
signal,
releaseOnClose: false,
});
const digesting = new Transform({
transform(chunk: Buffer, _encoding, callback) {
hash.update(chunk);
@@ -301,34 +342,50 @@ export class AsarArtifactReader implements ArtifactReader {
createWriteStream(snapshotPath, { flags: "wx", mode: 0o600 }),
signal === undefined ? {} : { signal },
);
const completedInput = inputHandle;
inputHandle = undefined;
await this.#closeOwnedHandle(completedInput);
this.#snapshot = { path: snapshotPath, sha256: hash.digest("hex") };
} catch (cause: unknown) {
const primary =
let failure =
signal?.aborted === true
? new ArtifactReaderFailure("cancelled", "ASAR operation cancelled", {
cause,
})
: asarFailure(this.path, "snapshot", cause);
if (inputHandle !== undefined) {
const failedInput = inputHandle;
inputHandle = undefined;
try {
await this.#closeOwnedHandle(failedInput);
} catch (cleanupCause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(cleanupCause, this.path),
);
}
}
try {
await this.#closeSnapshot();
if (this.#ownedHandles.size === 0) await this.#closeSnapshot();
} catch (cleanupCause: unknown) {
throw ArtifactReaderFailure.withCleanup(
primary,
failure,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
this.#snapshotRoot ?? this.path,
),
);
}
throw primary;
throw failure;
}
}
async #closeSnapshot(): Promise<void> {
await this.#closeOwnedHandles();
const root = this.#snapshotRoot;
if (root !== undefined) {
try {
await this.removeSnapshot(root);
await (this.options.removeSnapshot ?? defaultRemoveSnapshot)(root);
if (this.#snapshotRoot === root) this.#snapshotRoot = undefined;
} catch (cause: unknown) {
throw ArtifactReaderFailure.withCleanup(cause, {
@@ -386,7 +443,7 @@ export class AsarArtifactReader implements ArtifactReader {
signal?: AbortSignal,
): Promise<Readable> {
const unpackedRoot = `${this.path}.unpacked`;
let handle: UnpackedFileHandle | undefined;
let handle: SharedAsarFileHandle | undefined;
try {
const rootMetadata = await lstat(unpackedRoot);
if (!rootMetadata.isDirectory() || rootMetadata.isSymbolicLink())
@@ -413,15 +470,12 @@ export class AsarArtifactReader implements ArtifactReader {
"path",
`ASAR unpacked entry is not a regular file: ${entry.path}`,
);
handle = await this.openUnpackedFile(
handle = await (this.options.openUnpackedFile ?? open)(
canonical,
constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK,
);
const openedHandle = handle;
this.#unpackedHandles.set(openedHandle, {
path: canonical,
owner: new OwnedFileHandle(openedHandle),
});
this.#adoptHandle(openedHandle, canonical);
const openedMetadata = await openedHandle.stat();
if (
!openedMetadata.isFile() ||
@@ -441,30 +495,28 @@ export class AsarArtifactReader implements ArtifactReader {
);
}
abortIfNeeded(signal);
const source = readFileHandleChunks(openedHandle, {
start: 0,
});
source.once("close", () => {
void this.#closeUnpackedHandle(openedHandle).catch(() => undefined);
});
return readValidatedAsarEntry(
const source = this.#ownedReadStream(openedHandle, { start: 0 });
handle = undefined;
const output = readValidatedAsarEntry(
source,
undefined,
entry.path,
this.path,
signal,
);
this.#setOwnedOutput(openedHandle, output);
return output;
} catch (cause: unknown) {
let failure = asarUnpackedEntryFailure(this.path, entry, cause);
if (handle !== undefined && this.#unpackedHandles.has(handle)) {
if (handle !== undefined && this.#ownedHandles.has(handle)) {
try {
await this.#closeUnpackedHandle(handle);
await this.#closeOwnedHandle(handle);
} catch (cleanupCause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure,
ArtifactReaderFailure.cleanupObservation(
cleanupCause,
this.#unpackedHandles.get(handle)?.path ?? entry.path,
this.#ownedHandles.get(handle)?.path ?? entry.path,
),
);
}
@@ -473,11 +525,76 @@ export class AsarArtifactReader implements ArtifactReader {
}
}
async #closeUnpackedHandle(handle: UnpackedFileHandle): Promise<void> {
const owned = this.#unpackedHandles.get(handle);
async #closeOwnedHandle(handle: SharedAsarFileHandle): Promise<void> {
const owned = this.#ownedHandles.get(handle);
if (owned === undefined) return;
await owned.owner.close();
this.#unpackedHandles.delete(handle);
owned.output?.destroy();
owned.source?.destroy();
try {
await owned.owner.close();
} catch (cause: unknown) {
throw ArtifactReaderFailure.withCleanup(
cause,
ArtifactReaderFailure.cleanupObservation(cause, owned.path),
);
}
this.#ownedHandles.delete(handle);
}
async #closeOwnedHandles(): Promise<void> {
let failure: ArtifactReaderFailure | undefined;
for (const [handle, owned] of this.#ownedHandles) {
try {
await this.#closeOwnedHandle(handle);
} catch (cause: unknown) {
failure = ArtifactReaderFailure.withCleanup(
failure ?? cause,
ArtifactReaderFailure.cleanupObservation(cause, owned.path),
);
}
}
if (failure !== undefined) throw failure;
}
async #openOwned(path: string, flags: number): Promise<SharedAsarFileHandle> {
const handle = await (this.options.openFile ?? open)(path, flags);
this.#adoptHandle(handle, path);
return handle;
}
#adoptHandle(handle: SharedAsarFileHandle, path: string): void {
this.#ownedHandles.set(handle, {
path,
owner: new OwnedFileHandle(handle),
});
}
#ownedReadStream(
handle: SharedAsarFileHandle,
options: {
readonly start?: number;
readonly end?: number;
readonly signal?: AbortSignal | undefined;
readonly releaseOnClose?: boolean;
},
): Readable {
const { releaseOnClose, ...readOptions } = options;
const source = readFileHandleChunks(handle, readOptions);
const owned = this.#ownedHandles.get(handle);
if (owned === undefined)
throw new ArtifactReaderFailure("integrity", "ASAR handle has no owner");
owned.source = source;
owned.output = source;
if (releaseOnClose !== false)
source.once("close", () => {
void this.#closeOwnedHandle(handle).catch(() => undefined);
});
return source;
}
#setOwnedOutput(handle: SharedAsarFileHandle, output: Readable): void {
const owned = this.#ownedHandles.get(handle);
if (owned !== undefined) owned.output = output;
}
#resetArchiveState(): void {
@@ -491,16 +608,27 @@ type AsarEntryState = {
readonly metadata: AsarFileMetadata;
readonly entry: ArtifactEntry;
};
type UnpackedFileHandle = {
readonly fd: number;
type SharedAsarFileHandle = Pick<FileHandle, "fd" | "read" | "close"> & {
stat(): Promise<Stats>;
close(): Promise<void>;
read: FileHandle["read"];
};
type OpenAsarUnpackedFile = (
type OpenAsarFile = (
path: string,
flags: number,
) => Promise<UnpackedFileHandle>;
) => Promise<SharedAsarFileHandle>;
type OwnedAsarHandle = {
readonly path: string;
readonly owner: OwnedFileHandle<SharedAsarFileHandle>;
source?: Readable;
output?: Readable;
};
type AsarArtifactReaderOptions = {
readonly openFile?: OpenAsarFile;
readonly openUnpackedFile?: OpenAsarFile;
readonly removeSnapshot?: (path: string) => Promise<void>;
};
const defaultRemoveSnapshot = (path: string): Promise<void> =>
rm(path, { recursive: true, force: true });
const asarUnpackedEntryFailure = (
archivePath: string,
+2 -1
View File
@@ -1,3 +1,4 @@
import type { Stats } from "node:fs";
import type { FileHandle } from "node:fs/promises";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
@@ -7,7 +8,7 @@ export const MAX_ASAR_HEADER_BYTES = 16 * 1024 * 1024;
/** Admit both Chromium pickle extents before an upstream decoder allocates them. */
export const admitAsarHeader = async (
handle: FileHandle,
handle: Pick<FileHandle, "read"> & { stat(): Promise<Stats> },
): Promise<{
readonly archiveBytes: number;
readonly headerBytes: number;
+76 -20
View File
@@ -1,4 +1,4 @@
import { constants, type Stats } from "node:fs";
import { constants, type BigIntStats } from "node:fs";
import {
lstat,
open,
@@ -11,6 +11,7 @@ import { Readable } from "node:stream";
import {
ArtifactReaderFailure,
sameArtifactSourceIdentity,
type ArtifactEntry,
type ArtifactReader,
} from "./ArtifactReader.js";
@@ -95,9 +96,11 @@ export class DirectoryArtifactReader implements ArtifactReader {
if (child === null) break;
abortIfNeeded(signal);
const absolute = join(directory, child.name);
const metadata = await lstat(absolute).catch((cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
});
const metadata = await lstat(absolute, { bigint: true }).catch(
(cause: unknown) => {
throw directoryIoFailure("inspect entry", absolute, cause);
},
);
const path = relative(root, absolute).split(sep).join("/");
const entry = artifactEntry(path, absolute, metadata);
if (entry === undefined) continue;
@@ -158,18 +161,18 @@ export class DirectoryArtifactReader implements ArtifactReader {
const owner = new OwnedFileHandle(fileHandle);
this.#files.set(owner, entry.adapterKey);
try {
const observed = await owner.handle.stat().catch((cause: unknown) => {
throw directoryIoFailure(
"inspect opened entry",
entry.adapterKey,
cause,
);
});
const observed = await owner.handle
.stat({ bigint: true })
.catch((cause: unknown) => {
throw directoryIoFailure(
"inspect opened entry",
entry.adapterKey,
cause,
);
});
if (
!observed.isFile() ||
entry.sourceIdentity === undefined ||
observed.dev !== entry.sourceIdentity.device ||
observed.ino !== entry.sourceIdentity.inode
!sameArtifactSourceIdentity(entry.sourceIdentity, observed)
) {
throw new ArtifactReaderFailure(
"integrity",
@@ -178,7 +181,7 @@ export class DirectoryArtifactReader implements ArtifactReader {
}
abortIfNeeded(signal);
const source = Readable.from(
mapFileReadCancellation(owner.handle, signal),
mapStableFileRead(owner.handle, entry, signal),
{ objectMode: false },
);
source.once("close", () => {
@@ -256,7 +259,7 @@ export class DirectoryArtifactReader implements ArtifactReader {
const artifactEntry = (
path: string,
absolute: string,
metadata: Stats,
metadata: BigIntStats,
): ArtifactEntry | undefined => {
const kind = metadata.isSymbolicLink()
? "symlink"
@@ -269,7 +272,7 @@ const artifactEntry = (
return {
path,
kind,
declaredSize: kind === "file" ? metadata.size : null,
declaredSize: kind === "file" ? safeArtifactSize(metadata.size) : null,
compressedSize: null,
executable: hostExecutableMarking(absolute, metadata),
encrypted: false,
@@ -283,12 +286,24 @@ const artifactEntry = (
adapterKey: absolute,
...(kind === "file"
? {
sourceIdentity: { device: metadata.dev, inode: metadata.ino },
sourceIdentity: {
device: metadata.dev,
inode: metadata.ino,
size: metadata.size,
mode: metadata.mode,
mtimeNs: metadata.mtimeNs,
ctimeNs: metadata.ctimeNs,
},
}
: {}),
};
};
const safeArtifactSize = (size: bigint): number | null => {
if (size < 0n || size > BigInt(Number.MAX_SAFE_INTEGER)) return null;
return Number(size);
};
const directoryIoFailure = (
operation: string,
path: string,
@@ -313,16 +328,45 @@ const directoryIoFailure = (
const failureMessage = (cause: unknown): string =>
cause instanceof Error ? cause.message : String(cause);
const mapFileReadCancellation = async function* (
const mapStableFileRead = async function* (
handle: FileHandle,
entry: ArtifactEntry,
signal?: AbortSignal,
): AsyncGenerator<Buffer> {
let bytes = 0n;
try {
for await (const chunk of readFileHandleChunks(handle, { signal })) {
const expectedBytes = entry.sourceIdentity?.size;
if (expectedBytes === undefined)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry has no captured file state: ${entry.path}`,
);
for await (const chunk of readFileHandleChunks(handle, {
// Include one extra byte so a growing file cannot make this read unbounded.
end: expectedBytes,
signal,
})) {
abortIfNeeded(signal);
bytes += BigInt(chunk.length);
yield chunk;
}
abortIfNeeded(signal);
const [opened, current] = await Promise.all([
handle.stat({ bigint: true }),
lstat(entry.adapterKey, { bigint: true }),
]);
abortIfNeeded(signal);
if (
!opened.isFile() ||
!current.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, opened) ||
!sameArtifactSourceIdentity(entry.sourceIdentity, current) ||
bytes !== expectedBytes
)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry changed during read: ${entry.path}`,
);
} catch (cause: unknown) {
if (signal?.aborted === true)
throw new ArtifactReaderFailure(
@@ -330,6 +374,18 @@ const mapFileReadCancellation = async function* (
"Directory artifact read cancelled",
{ cause },
);
if (
cause instanceof Error &&
"code" in cause &&
["ENOENT", "ENOTDIR", "ELOOP"].includes(String(cause.code))
)
throw new ArtifactReaderFailure(
"integrity",
`Directory entry changed during read: ${entry.path}`,
{ cause },
);
if (!(cause instanceof ArtifactReaderFailure))
throw directoryIoFailure("validate read entry", entry.adapterKey, cause);
throw cause;
}
};
+84 -31
View File
@@ -125,6 +125,7 @@ export class NativeDmgArtifactReader implements ArtifactReader {
#ownershipUncertainty: string | undefined;
#closed = false;
#closePromise: Promise<void> | undefined;
readonly #activeDirectoryOperations = new Set<Promise<unknown>>();
constructor(
private readonly path: string,
@@ -141,18 +142,30 @@ export class NativeDmgArtifactReader implements ArtifactReader {
async *entries(signal?: AbortSignal): AsyncIterable<ArtifactEntry> {
await this.#ensureAttached(signal);
if (this.#directory === undefined)
const directory = this.#directory;
if (directory === undefined)
throw new ArtifactReaderFailure("unavailable", "DMG is not attached");
const prefix = basename(this.path);
for await (const entry of this.#directory.entries(signal))
yield { ...entry, path: `${prefix}/${entry.path}` };
const entries = directory.entries(signal)[Symbol.asyncIterator]();
try {
for (;;) {
if (this.#closed) return;
const next = await this.#trackDirectoryOperation(() => entries.next());
if (this.#closed) return;
if (next.done) return;
yield { ...next.value, path: `${prefix}/${next.value.path}` };
}
} finally {
await entries.return?.();
}
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
await this.#ensureAttached(signal);
if (this.#directory === undefined)
const directory = this.#directory;
if (directory === undefined)
throw new ArtifactReaderFailure("unavailable", "DMG is not attached");
return this.#directory.open(entry, signal);
return this.#trackDirectoryOperation(() => directory.open(entry, signal));
}
provenance(): readonly ArtifactCommand[] {
@@ -172,32 +185,9 @@ export class NativeDmgArtifactReader implements ArtifactReader {
async #closeOwnedResources(): Promise<void> {
await this.#initialization?.catch(() => undefined);
let detachFailure = await this.#reconcileAttachState();
let remainingDevices: string[] = [];
for (const device of [...this.#devices].reverse()) {
const failure = await this.#detach(device);
if (failure !== undefined) {
remainingDevices.push(device);
detachFailure ??= failure;
}
}
// APFS container detach may briefly leave its backing image listed.
for (const delayMs of DETACH_SETTLE_DELAYS_MS) {
if (remainingDevices.length === 0) break;
await (this.host.delay ?? delay)(delayMs);
const stillAttached: string[] = [];
for (const device of remainingDevices) {
if (!(await this.#isAttached(device))) continue;
const failure = await this.#detach(device);
if (failure !== undefined) {
stillAttached.push(device);
detachFailure = failure;
}
}
remainingDevices = stillAttached;
}
if (remainingDevices.length === 0) detachFailure = undefined;
this.#devices = remainingDevices.reverse();
await Promise.allSettled(this.#activeDirectoryOperations);
await this.#closeMountedDirectory();
let detachFailure = await this.#detachOwnedDevices();
if (
this.#devices.length === 0 &&
this.#mountRoot !== undefined &&
@@ -242,6 +232,55 @@ export class NativeDmgArtifactReader implements ArtifactReader {
);
}
async #detachOwnedDevices(): Promise<unknown> {
let detachFailure = await this.#reconcileAttachState();
let remainingDevices: string[] = [];
for (const device of [...this.#devices].reverse()) {
const failure = await this.#detach(device);
if (failure !== undefined) {
remainingDevices.push(device);
detachFailure ??= failure;
}
}
// APFS container detach may briefly leave its backing image listed.
for (const delayMs of DETACH_SETTLE_DELAYS_MS) {
if (remainingDevices.length === 0) break;
await (this.host.delay ?? delay)(delayMs);
const stillAttached: string[] = [];
for (const device of remainingDevices) {
if (!(await this.#isAttached(device))) continue;
const failure = await this.#detach(device);
if (failure !== undefined) {
stillAttached.push(device);
detachFailure = failure;
}
}
remainingDevices = stillAttached;
}
if (remainingDevices.length === 0) detachFailure = undefined;
this.#devices = remainingDevices.reverse();
return detachFailure;
}
async #closeMountedDirectory(): Promise<void> {
if (this.#directory !== undefined) {
try {
await this.#directory.close();
this.#directory = undefined;
} catch (cause: unknown) {
const mountRoot = this.#mountRoot ?? this.path;
throw ArtifactReaderFailure.withCleanup(cause, {
reason: `DMG mounted directory remains owned: ${failureMessage(cause)}`,
resources: [
`DMG mounted directory at ${mountRoot}`,
`DMG mount root ${mountRoot}`,
...this.#devices.map((device) => `DMG device ${device}`),
],
});
}
}
}
/** Detach one device; return the failure only while it remains attached. */
async #detach(device: string): Promise<unknown> {
try {
@@ -294,6 +333,20 @@ export class NativeDmgArtifactReader implements ArtifactReader {
throw new ArtifactReaderFailure("unavailable", "DMG reader is closed");
}
async #trackDirectoryOperation<Result>(
operation: () => Promise<Result>,
): Promise<Result> {
if (this.#closed)
throw new ArtifactReaderFailure("unavailable", "DMG reader is closed");
const running = Promise.resolve().then(operation);
this.#activeDirectoryOperations.add(running);
try {
return await running;
} finally {
this.#activeDirectoryOperations.delete(running);
}
}
async #attach(signal?: AbortSignal): Promise<void> {
await runChecked(this.host, ["verify", this.path], signal);
this.#provenance.push(command(["verify", this.path], ["read"]));
@@ -1,7 +1,10 @@
import { createHash } from "node:crypto";
import { lstat } from "node:fs/promises";
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
import { ArtifactReaderFailure } from "../ArtifactReader.js";
import {
ArtifactReaderFailure,
sameArtifactSourceIdentity,
} from "../ArtifactReader.js";
import { analyzeInterfaceBuilderBundle } from "./InterfaceBuilderAnalysis.js";
import {
appleAssetCatalogInputSchema,
@@ -136,22 +139,20 @@ const inspectAssetCatalogEntry = async (input: {
}) => {
const { entry, reader, signal } = input;
const sha256 = await digestBounded(reader, entry, signal);
const before = await lstat(entry.adapterKey);
const before = await lstat(entry.adapterKey, { bigint: true });
if (
entry.sourceIdentity === undefined ||
before.dev !== entry.sourceIdentity.device ||
before.ino !== entry.sourceIdentity.inode
!before.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, before)
)
throw new ArtifactReaderFailure(
"integrity",
`Asset catalog changed before inspection: ${entry.path}`,
);
const output = await input.runAssetUtil(entry.adapterKey, signal);
const after = await lstat(entry.adapterKey);
const after = await lstat(entry.adapterKey, { bigint: true });
if (
before.dev !== after.dev ||
before.ino !== after.ino ||
before.size !== after.size ||
!after.isFile() ||
!sameArtifactSourceIdentity(entry.sourceIdentity, after) ||
(await digestBounded(reader, entry, signal)) !== sha256
)
throw new ArtifactReaderFailure(
@@ -526,12 +526,16 @@ const CPU_TYPE_X86 = 7;
const CPU_TYPE_ARM = 12;
const CPU_SUBTYPE_MASK = 0xff000000;
const CPU_SUBTYPE_ARM64E = 2;
const CPU_SUBTYPE_ARM64E_X1 = 12;
const CPU_SUBTYPE_X86_64_H = 8;
const architectureName = (cpuType: number, cpuSubtype: number): string => {
const subtype = cpuSubtype & ~CPU_SUBTYPE_MASK;
if (cpuType === (CPU_TYPE_ARM | CPU_ARCH_ABI64))
return subtype === CPU_SUBTYPE_ARM64E ? "arm64e" : "arm64";
// Both pointer-authentication subtypes belong to the arm64e family.
return subtype === CPU_SUBTYPE_ARM64E || subtype === CPU_SUBTYPE_ARM64E_X1
? "arm64e"
: "arm64";
if (cpuType === (CPU_TYPE_ARM | CPU_ARCH_ABI64_32)) return "arm64_32";
if (cpuType === (CPU_TYPE_X86 | CPU_ARCH_ABI64))
return subtype === CPU_SUBTYPE_X86_64_H ? "x86_64h" : "x86_64";
+5 -3
View File
@@ -1,4 +1,4 @@
import type { Stats } from "node:fs";
import type { BigIntStats, Stats } from "node:fs";
import { extname } from "node:path";
/** Extensions Windows launches directly through CreateProcess or cmd.exe. */
@@ -13,10 +13,12 @@ const windowsLaunchableExtensions = new Set([".bat", ".cmd", ".com", ".exe"]);
*/
export const hostExecutableMarking = (
path: string,
metadata: Pick<Stats, "mode" | "isFile">,
metadata: Pick<Stats | BigIntStats, "mode" | "isFile">,
platform: NodeJS.Platform = process.platform,
): boolean =>
platform === "win32"
? metadata.isFile() &&
windowsLaunchableExtensions.has(extname(path).toLowerCase())
: (metadata.mode & 0o111) !== 0;
: typeof metadata.mode === "bigint"
? (metadata.mode & 0o111n) !== 0n
: (metadata.mode & 0o111) !== 0;
@@ -18,6 +18,7 @@ import { streamChunkToBuffer } from "../StreamBytes.js";
import { hashReadable } from "../ArtifactHash.js";
import type { ArtifactOccurrence } from "../../domain/artifactGraph.js";
import type { ArtifactInventorySnapshot } from "../../domain/artifactInventorySnapshot.js";
import { AnalysisInputError } from "../../domain/analysisErrorCore.js";
import type {
JavaScriptArtifactFileKind,
@@ -47,6 +48,7 @@ interface ReadContext {
readonly files: JavaScriptArtifactFile[];
readonly containers: JavaScriptArtifactContainer[];
readonly signal: AbortSignal | undefined;
readonly selectedPaths: ReadonlyMap<string, number> | undefined;
textBytes: number;
invalidUtf8: number;
}
@@ -57,13 +59,49 @@ interface ReadTextInput {
readonly expected: ExpectedFile;
}
/** Read all relevant textual entries through an already-inventoried reader. */
/** Verify inventoried entries; acquire text only inside the caller's scope. */
export const readJavaScriptArtifactFiles = async (
reader: ArtifactReader,
snapshot: ArtifactInventorySnapshot,
resources: ArtifactResourceScope,
signal?: AbortSignal,
sourcePaths?: readonly string[],
): Promise<JavaScriptArtifactFileSet> => {
const inventory = expectedInventory(snapshot);
const expected = inventory.files;
const selectedPaths =
sourcePaths === undefined
? undefined
: new Map([...new Set(sourcePaths)].map((path, index) => [path, index]));
const issues = (sourcePaths ?? []).flatMap((path, index) =>
expected.has(path) || inventory.unavailableFiles.has(path)
? []
: [
{
path: ["source_paths", index],
reason: "invalid_value" as const,
message:
"Select an exact relevant file path from the artifact inventory; no globs or path aliases.",
},
],
);
if (issues.length > 0)
throw new AnalysisInputError(
"analyze_javascript_application",
undefined,
issues,
);
const unavailablePath = sourcePaths?.find((path) =>
inventory.unavailableFiles.has(path),
);
if (unavailablePath !== undefined)
throw new ArtifactReaderFailure(
"unavailable",
`Selected inventoried source bytes are unavailable: ${unavailablePath}`,
{
partialObservation: { kind: "artifact-inventory", inventory: snapshot },
},
);
if (reader instanceof AsarArtifactReader) {
const root = snapshot.nodes.find(
({ artifact_id }) => artifact_id === snapshot.manifest.root_artifact_id,
@@ -79,8 +117,6 @@ export const readJavaScriptArtifactFiles = async (
signal,
);
}
const inventory = expectedInventory(snapshot);
const expected = inventory.files;
const context: ReadContext = {
resources,
expected,
@@ -89,13 +125,17 @@ export const readJavaScriptArtifactFiles = async (
files: [],
containers: [],
signal,
selectedPaths,
textBytes: 0,
invalidUtf8: 0,
};
await visitReader(reader, "", snapshot.manifest.root_sha256, context);
const files = context.files.sort((left, right) =>
compareUnicodeCodePoints(left.path, right.path),
);
const files = context.files.sort((left, right) => {
const leftPriority = selectedPaths?.get(left.path) ?? Infinity;
const rightPriority = selectedPaths?.get(right.path) ?? Infinity;
if (leftPriority !== rightPriority) return leftPriority - rightPriority;
return compareUnicodeCodePoints(left.path, right.path);
});
assertExpectedFilesWereVisited(expected, files);
return {
files,
@@ -104,6 +144,17 @@ export const readJavaScriptArtifactFiles = async (
),
text_bytes_read: context.textBytes,
invalid_utf8_files: context.invalidUtf8,
...(sourcePaths === undefined
? {}
: {
source_selection: {
requested_paths: [...sourcePaths],
unselected_files: files.filter(
({ text }) =>
!text.included && text.reason === "outside-analysis-scope",
).length,
},
}),
};
};
@@ -251,6 +302,19 @@ const readText = async (
context: ReadContext,
input: ReadTextInput,
): Promise<JavaScriptArtifactFile["text"]> => {
if (
context.selectedPaths !== undefined &&
!context.selectedPaths.has(input.expected.path)
) {
// Keep the inventory/acquisition identity check without allocating or
// decoding an unrelated source's text.
await verifyEntryBytes(
await input.reader.open(input.entry, context.signal),
input.expected,
context.signal,
);
return { included: false, reason: "outside-analysis-scope" };
}
if (input.expected.kind === "native-addon") {
await verifyEntryBytes(
await input.reader.open(input.entry, context.signal),
@@ -311,16 +375,23 @@ const expectedInventory = (
): {
readonly files: ReadonlyMap<string, ExpectedFile>;
readonly containers: ReadonlyMap<string, ExpectedContainer>;
readonly unavailableFiles: ReadonlySet<string>;
} => {
const nodes = new Map(snapshot.nodes.map((node) => [node.artifact_id, node]));
const files = new Map<string, ExpectedFile>();
const containers = new Map<string, ExpectedContainer>();
const unavailableFiles = new Set<string>();
for (const occurrence of snapshot.occurrences) {
if (occurrence.logical_path === ".") continue;
const isNestedAsar =
occurrence.entry_kind === "file" &&
occurrence.logical_path.toLowerCase().endsWith(".asar");
if (occurrence.artifact_id === null) {
if (
occurrence.entry_kind === "file" &&
relevantKind(occurrence.logical_path) !== undefined
)
unavailableFiles.add(occurrence.logical_path);
if (isNestedAsar)
containers.set(occurrence.logical_path, {
hash_status: occurrence.hash_status,
@@ -358,7 +429,7 @@ const expectedInventory = (
kind,
});
}
return { files, containers };
return { files, containers, unavailableFiles };
};
const readAll = async (
-62
View File
@@ -1,62 +0,0 @@
import {
lstat,
mkdtemp,
open,
rm,
symlink,
unlink,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { expect, it } from "vitest";
import { readStableArtifact } from "./readStableArtifact.js";
const checkChange = async (
stage: "before-open" | "after-read" | "symlink-replacement",
): Promise<void> => {
const root = await mkdtemp(join(tmpdir(), "rea-stable-read-"));
const path = join(root, "selected");
await writeFile(path, "original evidence");
let inspected = 0;
try {
const reading = readStableArtifact(path, 1024, undefined, {
async lstat(selected) {
inspected++;
if (stage === "after-read" && inspected === 2) await unlink(selected);
const stat = await lstat(selected, { bigint: true });
if (inspected === 1 && stage !== "after-read") {
await unlink(selected);
if (stage === "symlink-replacement")
await symlink("missing", selected);
}
return stat;
},
open,
});
await expect(reading).rejects.toMatchObject({ reason: "integrity" });
} finally {
await rm(root, { recursive: true, force: true });
}
};
it.each(["before-open", "after-read"] as const)(
"retains an actual post-validation %s change as integrity failure",
checkChange,
);
it.skipIf(process.platform === "win32")(
"retains a post-validation symlink replacement as an integrity failure",
() => checkChange("symlink-replacement"),
);
it("keeps initially missing inputs distinguishable from later changes", async () => {
const root = await mkdtemp(join(tmpdir(), "rea-stable-missing-"));
try {
await expect(
readStableArtifact(join(root, "missing"), 1024),
).rejects.toMatchObject({ code: "ENOENT" });
} finally {
await rm(root, { recursive: true, force: true });
}
});
+29 -15
View File
@@ -226,10 +226,17 @@ export const handleRequestWillBeSent = (
preserveMalformedRedirectEvidence(state, requestId, previous)
)
return;
if (request === undefined) {
state.completeness.exclude("network_requests", "invalid_protocol_value");
state.network.delete(requestId);
state.networkRequestTimestamps.delete(requestId);
const method = cdpStringValue(request?.method);
const initiator = recordValue(params.initiator);
const initiatorType = cdpStringValue(initiator?.type);
if (
request === undefined ||
method === undefined ||
method.length === 0 ||
initiatorType === undefined ||
initiatorType.length === 0
) {
preserveMalformedRedirectEvidence(state, requestId, previous);
return;
}
const sanitized = allowedSanitizedUrl(request.url, state.allowedOrigins);
@@ -273,6 +280,12 @@ export const handleRequestWillBeSent = (
state.networkRequestTimestamps.delete(requestId);
return;
} else {
const encodedDataLength = numberValue(redirectResponse.encodedDataLength);
if (encodedDataLength === undefined || encodedDataLength < 0)
state.completeness.exclude(
"network_requests",
"invalid_protocol_value",
);
redirects.push({
url: previous.url,
response_url: responseUrl.url,
@@ -280,10 +293,10 @@ export const handleRequestWillBeSent = (
resource_type: previous.resource_type,
status: numberValue(redirectResponse.status) ?? null,
mime_type: cdpStringValue(redirectResponse.mimeType) ?? null,
encoded_data_length: (() => {
const length = numberValue(redirectResponse.encodedDataLength);
return length === undefined ? null : Math.max(0, length);
})(),
encoded_data_length:
encodedDataLength === undefined || encodedDataLength < 0
? null
: encodedDataLength,
request_timestamp:
state.networkRequestTimestamps.get(requestId) ?? null,
// CDP reports redirectResponse on the next requestWillBeSent event.
@@ -292,7 +305,6 @@ export const handleRequestWillBeSent = (
});
}
}
const initiator = recordValue(params.initiator);
const initiatorFrame = initiatorLocation(initiator);
const rawInitiatorUrl = cdpStringValue(initiatorFrame?.url);
const initiatorUrl = allowedSanitizedUrl(
@@ -308,14 +320,14 @@ export const handleRequestWillBeSent = (
request_id: requestId,
url: sanitized.url,
origin: sanitized.origin ?? "",
method: cdpStringValue(request.method) ?? "GET",
method,
resource_type: cdpStringValue(params.type) ?? null,
status: null,
mime_type: null,
encoded_data_length: null,
redirects,
initiator: {
type: cdpStringValue(initiator?.type) ?? "other",
type: initiatorType,
url: initiatorUrl?.url ?? null,
line: integerOrNull(initiatorFrame?.lineNumber),
column: integerOrNull(initiatorFrame?.columnNumber),
@@ -411,12 +423,14 @@ export const handleLoadingFinished = (
if (state.malformedRedirectRequestIds.has(requestId)) return;
const current = state.network.get(requestId);
if (current === undefined) return;
const encodedDataLength = numberValue(params.encodedDataLength);
if (encodedDataLength === undefined || encodedDataLength < 0) {
state.completeness.exclude("network_requests", "invalid_protocol_value");
return;
}
state.network.set(requestId, {
...current,
encoded_data_length: Math.max(
0,
numberValue(params.encodedDataLength) ?? 0,
),
encoded_data_length: encodedDataLength,
});
};
+1 -1
View File
@@ -20,10 +20,10 @@ import type {
import { webSourceMapsSchema } from "../domain/webBundleAnalysis.js";
import { createWebTextArtifact } from "../domain/webContentArtifact.js";
import { safeParseJson } from "../domain/safeJson.js";
import { sourceMapJsonText } from "../domain/sourceMapEnvelope.js";
import { jsonParts } from "../domain/jsonSerialization.js";
import {
inspectSourceMapValue,
sourceMapJsonText,
SourceMapFormatFailure,
type SourceMapLeaf,
} from "../javascript/sourceMaps/SourceMapFormat.js";
+129 -29
View File
@@ -7,48 +7,148 @@ const input = {
importMap: null,
specifiers: ["./dep.js"],
};
describe("optional native resolution acquisition", () => {
it("awaits owned cleanup when cancellation arrives during acquisition", async () => {
const controller = new AbortController();
let closed = false;
describe("native module resolver cleanup ownership", () => {
it.each([false, true])(
"awaits owned cleanup when acquisition is cancelled (cleanup fails: %s)",
async (cleanupFails) => {
const controller = new AbortController();
let closed = false;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() => {
controller.abort();
return Promise.resolve({
version: () => "fixture",
newContext: () => {
throw new Error("must not acquire context");
},
close: () => {
closed = true;
if (cleanupFails)
return Promise.reject(new Error("fixture cleanup refusal"));
return Promise.resolve();
},
});
},
);
const result = await provider.resolve(input, {
signal: controller.signal,
});
expect(closed).toBe(true);
if (result.ok) throw new Error("expected cancelled result");
const projected = projectAnalysisError(result.error);
expect(projected.code).toBe(
cleanupFails ? "cleanup_incomplete" : "cancelled",
);
if (cleanupFails) {
expect(projected.details).toMatchObject({
diagnostics: { primary_error: { code: "cancelled" } },
});
}
},
);
it("returns cleanup_incomplete when owned cleanup fails after an engine error", async () => {
let launches = 0;
let closes = 0;
let cleanupAllowed = false;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() => {
controller.abort();
launches += 1;
return Promise.resolve({
version: () => "fixture",
newContext: () => {
throw new Error("must not acquire context");
},
close: () => {
closed = true;
return Promise.resolve();
},
});
},
);
const result = await provider.resolve(input, { signal: controller.signal });
expect(closed).toBe(true);
if (result.ok) throw new Error("expected cancelled result");
expect(result.error._tag).toBe("AnalysisCancelledError");
});
it("returns cleanup_incomplete when owned cleanup fails after an engine error", async () => {
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
() =>
Promise.resolve({
version: () => "fixture",
newContext: () =>
Promise.reject(new Error("fixture context failure")),
close: () =>
Promise.reject(new Error("fixture owned cleanup refusal")),
}),
close: async () => {
closes += 1;
if (!cleanupAllowed)
throw new Error("fixture owned cleanup refusal");
},
});
},
);
const result = await provider.resolve(input);
if (result.ok) throw new Error("expected cleanup failure");
expect(projectAnalysisError(result.error).code).toBe("cleanup_incomplete");
expect(result.error.cleanupIncomplete).toBe(true);
expect(projectAnalysisError(result.error).details).toMatchObject({
diagnostics: {
primary_error: {
details: {
diagnostics: { error_message: "fixture context failure" },
},
},
cleanup_error: {
details: { diagnostics: { reason: "fixture owned cleanup refusal" } },
},
},
});
const blocked = await provider.resolve(input);
expect(blocked.ok).toBe(false);
if (!blocked.ok) expect(blocked.error.cleanupIncomplete).toBe(true);
expect(launches).toBe(1);
await expect(provider.close()).rejects.toMatchObject({
cleanupIncomplete: true,
});
cleanupAllowed = true;
await provider.close();
await provider.close();
expect(closes).toBe(4);
expect(launches).toBe(1);
});
it("awaits acquisition and cleanup while refusing queued work during shutdown", async () => {
let acquired: () => void = () => {
throw new Error("acquisition gate missing");
};
const acquiring = new Promise<void>((resolve) => {
acquired = resolve;
});
let release: () => void = () => {
throw new Error("release gate missing");
};
const released = new Promise<void>((resolve) => {
release = resolve;
});
let launches = 0;
let closes = 0;
const provider = new NativeModuleResolver(
{ REA_BROWSER_EXECUTABLE: "/browser" },
async () => {
launches += 1;
acquired();
await released;
return {
version: () => "fixture",
newContext: () =>
Promise.reject(new Error("fixture context failure")),
close: async () => {
closes += 1;
},
};
},
);
const first = provider.resolve(input);
await acquiring;
const queued = provider.resolve(input);
const closing = provider.close();
let closed = false;
void closing.then(() => {
closed = true;
});
await Promise.resolve();
expect(closed).toBe(false);
release();
await first;
const blocked = await queued;
expect(blocked.ok).toBe(false);
await closing;
expect(closed).toBe(true);
expect([launches, closes]).toEqual([1, 1]);
expect((await provider.resolve(input)).ok).toBe(false);
});
});
describe("optional native resolution acquisition", () => {
it("does not load the browser engine when the selected operation lacks its configuration", async () => {
const provider = new NativeModuleResolver({}, () => {
throw new Error("must not launch");
+242 -147
View File
@@ -7,6 +7,7 @@ import type {
WebModuleResolutionBatch,
} from "../../application/WebModulePorts.js";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
import {
AnalysisCancelledError,
AnalysisCapabilityUnavailableError,
@@ -16,6 +17,7 @@ import {
import { BrowserObservationError } from "../../domain/browserObservationError.js";
import { jsonObjectSchema } from "../../domain/jsonValue.js";
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
import { err, ok, type Result } from "../../domain/result.js";
import { webModuleResolutionSchema } from "../../domain/webModuleTrace.js";
import { ProviderStartupDeadline } from "../../process/ProviderDeadline.js";
@@ -32,26 +34,49 @@ const reportSchema = z.strictObject({
),
});
type ModuleResolverBrowser = Pick<Browser, "newContext" | "version" | "close">;
/** Production launch seam: only the adapter knows the browser API. */
export type ModuleResolverLauncher = (
options: LaunchOptions,
) => Promise<Pick<Browser, "newContext" | "version" | "close">>;
) => Promise<ModuleResolverBrowser>;
const launch: ModuleResolverLauncher = async (options) =>
(await import("playwright-core")).chromium.launch(options);
/** Use native import.meta.resolve in an owned context with locally fulfilled requests. */
export class NativeModuleResolver implements WebModuleResolutionPort {
#tail: Promise<void> = Promise.resolve();
#closing = false;
#closePromise: Promise<void> | undefined;
#pendingBrowser: ModuleResolverBrowser | undefined;
constructor(
readonly environment: Readonly<Record<string, string | undefined>>,
readonly launcher: ModuleResolverLauncher = launch,
) {}
async resolve(
resolve(
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
options?: ExecutionOptions,
): ReturnType<WebModuleResolutionPort["resolve"]> {
if (this.#closing) return Promise.resolve(err(this.#closedError()));
const operation = this.#tail.then(() =>
this.#closing ? err(this.#closedError()) : this.#resolve(input, options),
);
this.#tail = operation.then(
() => undefined,
() => undefined,
);
return operation;
}
async #resolve(
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
options?: ExecutionOptions,
): ReturnType<WebModuleResolutionPort["resolve"]> {
if (options?.signal?.aborted === true)
return err(new AnalysisCancelledError(OPERATION));
const pending = await this.#retryCleanup();
if (pending !== undefined) return err(pending);
const executable = this.environment.REA_BROWSER_EXECUTABLE;
if (executable === undefined || !isAbsolute(executable))
return err(
@@ -66,11 +91,16 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
),
);
const deadline = new ProviderStartupDeadline(TIMEOUT_MS, options?.signal);
let browser: Pick<Browser, "newContext" | "version" | "close"> | undefined;
let browser: ModuleResolverBrowser | undefined;
let cleanup: Promise<void> | undefined;
const close = (): Promise<void> => {
if (cleanup === undefined && browser !== undefined)
cleanup = browser.close();
if (cleanup === undefined && browser !== undefined) {
const owned = browser;
cleanup = Promise.resolve().then(async () => {
await owned.close();
this.#pendingBrowser = undefined;
});
}
return cleanup ?? Promise.resolve();
};
const onAbort = (): void => {
@@ -82,6 +112,7 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
deadline.signal.addEventListener("abort", onAbort, { once: true });
let result: Result<WebModuleResolutionBatch, AnalysisError>;
try {
deadline.signal.throwIfAborted();
browser = await this.launcher({
executablePath: executable,
headless: true,
@@ -99,142 +130,9 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
: []),
],
});
this.#pendingBrowser = browser;
deadline.signal.throwIfAborted();
const context = await browser.newContext({
serviceWorkers: "block",
acceptDownloads: false,
});
const page = await context.newPage();
const diagnostics: string[] = [];
let diagnosticBytes = 0;
page.on("console", (message) => {
const text = message.text();
diagnosticBytes += Buffer.byteLength(text);
if (diagnosticBytes <= 1024 * 1024)
diagnostics.push(`${message.type()}: ${text}`);
});
const requests: {
url: string;
resource_type: string;
effect: "fulfilled" | "blocked";
}[] = [];
const importerUrl = new URL(input.importerUrl).href;
const documentUrl = new URL("/.rea-module-resolver/document", importerUrl)
.href;
const transportUrl = new URL(importerUrl);
transportUrl.hash = "";
let stubSent = false;
await context.route("**/*", async (route) => {
const request = route.request();
if (
request.resourceType() === "document" &&
request.url() === documentUrl.split("#")[0]
) {
requests.push({
url: request.url(),
resource_type: "document",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/html",
body: `<!doctype html>${input.importMap === null ? "" : `<base href="${htmlAttribute(new URL(input.importMap.baseUrl).href)}"><script type="importmap">${scriptJson(input.importMap.value)}</script>`}`,
});
} else if (
!stubSent &&
request.resourceType() === "script" &&
request.url() === transportUrl.href
) {
stubSent = true;
requests.push({
url: request.url(),
resource_type: "script",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/javascript",
headers: { "Access-Control-Allow-Origin": "*" },
body: `if (typeof import.meta.resolve !== "function") {globalThis.__reaModuleUnsupported = true;} else {globalThis.__reaModuleResult = {importer_url: import.meta.url, entries: ${scriptJson(input.specifiers)}.map(specifier => {try {return {specifier, resolution: {state: "resolved", url: import.meta.resolve(specifier)}}} catch (error) {return {specifier, resolution: {state: "rejected", message: String(error)}}}})};}`,
});
} else {
requests.push({
url: request.url(),
resource_type: request.resourceType(),
effect: "blocked",
});
await route.abort();
}
});
await page.goto(documentUrl, {
timeout: Math.max(1, deadline.remainingMs()),
});
await page.evaluate(
`{const script = document.createElement("script"); script.type = "module"; script.src = ${scriptJson(importerUrl)}; script.onerror = () => {globalThis.__reaModuleFailure = "Trusted resolver stub could not be loaded in the selected context.";}; document.head.append(script);}`,
);
await page.waitForFunction(
() =>
Reflect.has(globalThis, "__reaModuleResult") ||
Reflect.has(globalThis, "__reaModuleUnsupported") ||
Reflect.has(globalThis, "__reaModuleFailure"),
undefined,
{ timeout: Math.max(1, deadline.remainingMs()) },
);
const unsupported: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleUnsupported") as unknown,
);
if (unsupported === true)
throw new AnalysisCapabilityUnavailableError(
"native-module-resolver",
OPERATION,
"native_resolution_unsupported",
{
userMessage: `Configured Chromium ${browser.version()} does not supply import.meta.resolve; use a browser supporting native module resolution.`,
},
);
const raw: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleResult") as unknown,
);
const failure: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleFailure") as unknown,
);
if (typeof failure === "string")
throw new AnalysisOutputError(
OPERATION,
`${failure} Native diagnostics: ${diagnostics.join("\n")}`,
);
const parsed = reportSchema.safeParse(raw);
if (
!parsed.success ||
parsed.data.importer_url !== importerUrl ||
parsed.data.entries.length !== input.specifiers.length ||
parsed.data.entries.some(
(entry, index) => entry.specifier !== input.specifiers[index],
)
)
throw new AnalysisOutputError(
OPERATION,
"Native resolver report has invalid shape or does not match the selected importer/specifiers.",
);
if (diagnosticBytes > 1024 * 1024)
throw new BrowserObservationError(OPERATION, "payload_limit");
deadline.signal.throwIfAborted();
const engine = {
id: "chromium-native-module-resolver",
version: browser.version(),
};
result = ok({
engine,
resolutions: parsed.data.entries.map((entry) => entry.resolution),
diagnostics,
rawResult: jsonObjectSchema.parse({
native_report: parsed.data,
requests,
diagnostics,
engine,
executable_path: executable,
}),
});
result = ok(await resolveInBrowser(browser, input, deadline, executable));
} catch (cause: unknown) {
result = err(
deadline.interruption === "cancelled"
@@ -257,20 +155,217 @@ export class NativeModuleResolver implements WebModuleResolutionPort {
} finally {
deadline.signal.removeEventListener("abort", onAbort);
deadline.dispose();
try {
await close();
} catch (cause: unknown) {
result = err(
new BrowserObservationError(OPERATION, "cleanup_failed", { cause }),
);
}
}
try {
await close();
} catch (cause: unknown) {
const failure = cleanupFailure(
cause,
result.ok ? result.value.rawResult : undefined,
);
return err(
result.ok
? failure
: analysisErrorWithCleanupFailure(result.error, failure, OPERATION),
);
}
if (result.ok && options?.signal?.aborted)
return err(new AnalysisCancelledError(OPERATION));
return result;
}
async close(): Promise<void> {
this.#closing = true;
this.#closePromise ??= this.#tail
.then(async () => {
const failure = await this.#retryCleanup();
if (failure !== undefined) throw failure;
})
.catch((cause: unknown) => {
this.#closePromise = undefined;
throw cause;
});
return this.#closePromise;
}
#closedError(): AnalysisError {
return new BrowserObservationError(OPERATION, "cleanup_failed", {
detail: "Native module resolver is closing.",
});
}
async #retryCleanup(): Promise<AnalysisError | undefined> {
const browser = this.#pendingBrowser;
if (browser === undefined) return undefined;
try {
await browser.close();
this.#pendingBrowser = undefined;
return undefined;
} catch (cause: unknown) {
return cleanupFailure(cause);
}
}
}
const resolveInBrowser = async (
browser: ModuleResolverBrowser,
input: Parameters<WebModuleResolutionPort["resolve"]>[0],
deadline: ProviderStartupDeadline,
executable: string,
): Promise<WebModuleResolutionBatch> => {
const context = await browser.newContext({
serviceWorkers: "block",
acceptDownloads: false,
});
const page = await context.newPage();
const diagnostics: string[] = [];
let diagnosticBytes = 0;
page.on("console", (message) => {
const text = message.text();
diagnosticBytes += Buffer.byteLength(text);
if (diagnosticBytes <= 1024 * 1024)
diagnostics.push(`${message.type()}: ${text}`);
});
const requests: {
url: string;
resource_type: string;
effect: "fulfilled" | "blocked";
}[] = [];
const importerUrl = new URL(input.importerUrl).href;
const documentUrl = new URL("/.rea-module-resolver/document", importerUrl)
.href;
const transportUrl = new URL(importerUrl);
transportUrl.hash = "";
let stubSent = false;
await context.route("**/*", async (route) => {
const request = route.request();
if (
request.resourceType() === "document" &&
request.url() === documentUrl.split("#")[0]
) {
requests.push({
url: request.url(),
resource_type: "document",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/html",
body: `<!doctype html>${input.importMap === null ? "" : `<base href="${htmlAttribute(new URL(input.importMap.baseUrl).href)}"><script type="importmap">${scriptJson(input.importMap.value)}</script>`}`,
});
} else if (
!stubSent &&
request.resourceType() === "script" &&
request.url() === transportUrl.href
) {
stubSent = true;
requests.push({
url: request.url(),
resource_type: "script",
effect: "fulfilled",
});
await route.fulfill({
status: 200,
contentType: "text/javascript",
headers: { "Access-Control-Allow-Origin": "*" },
body: `if (typeof import.meta.resolve !== "function") {globalThis.__reaModuleUnsupported = true;} else {globalThis.__reaModuleResult = {importer_url: import.meta.url, entries: ${scriptJson(input.specifiers)}.map(specifier => {try {return {specifier, resolution: {state: "resolved", url: import.meta.resolve(specifier)}}} catch (error) {return {specifier, resolution: {state: "rejected", message: String(error)}}}})};}`,
});
} else {
requests.push({
url: request.url(),
resource_type: request.resourceType(),
effect: "blocked",
});
await route.abort();
}
});
await page.goto(documentUrl, {
timeout: Math.max(1, deadline.remainingMs()),
});
await page.evaluate(
`{const script = document.createElement("script"); script.type = "module"; script.src = ${scriptJson(importerUrl)}; script.onerror = () => {globalThis.__reaModuleFailure = "Trusted resolver stub could not be loaded in the selected context.";}; document.head.append(script);}`,
);
await page.waitForFunction(
() =>
Reflect.has(globalThis, "__reaModuleResult") ||
Reflect.has(globalThis, "__reaModuleUnsupported") ||
Reflect.has(globalThis, "__reaModuleFailure"),
undefined,
{ timeout: Math.max(1, deadline.remainingMs()) },
);
const unsupported: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleUnsupported") as unknown,
);
if (unsupported === true)
throw new AnalysisCapabilityUnavailableError(
"native-module-resolver",
OPERATION,
"native_resolution_unsupported",
{
userMessage: `Configured Chromium ${browser.version()} does not supply import.meta.resolve; use a browser supporting native module resolution.`,
},
);
const raw: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleResult") as unknown,
);
const failure: unknown = await page.evaluate(
() => Reflect.get(globalThis, "__reaModuleFailure") as unknown,
);
if (typeof failure === "string")
throw new AnalysisOutputError(
OPERATION,
`${failure} Native diagnostics: ${diagnostics.join("\n")}`,
);
const parsed = reportSchema.safeParse(raw);
if (
!parsed.success ||
parsed.data.importer_url !== importerUrl ||
parsed.data.entries.length !== input.specifiers.length ||
parsed.data.entries.some(
(entry, index) => entry.specifier !== input.specifiers[index],
)
)
throw new AnalysisOutputError(
OPERATION,
"Native resolver report has invalid shape or does not match the selected importer/specifiers.",
);
if (diagnosticBytes > 1024 * 1024)
throw new BrowserObservationError(OPERATION, "payload_limit");
deadline.signal.throwIfAborted();
const engine = {
id: "chromium-native-module-resolver",
version: browser.version(),
};
return {
engine,
resolutions: parsed.data.entries.map((entry) => entry.resolution),
diagnostics,
rawResult: jsonObjectSchema.parse({
native_report: parsed.data,
requests,
diagnostics,
engine,
executable_path: executable,
}),
};
};
const cleanupFailure = (
cause: unknown,
observation?: WebModuleResolutionBatch["rawResult"],
): ProviderCleanupError =>
new ProviderCleanupError(
"native-module-resolver",
["chromium-browser"],
{
reason: cause instanceof Error ? cause.message : String(cause),
...(observation === undefined
? {}
: { partial_observation: observation }),
},
{ cause, operation: OPERATION },
);
const scriptJson = (value: unknown): string =>
JSON.stringify(value).replaceAll("<", "\\u003c");
+22 -4
View File
@@ -15,6 +15,7 @@ import type { CliInstance } from "./types.js";
import { runCliJavaScriptApplicationAnalysis } from "./javascriptApplicationAnalysis.js";
import { withCommandCancellation } from "./commandCancellation.js";
import { javascriptApplicationOptions } from "../cliObservationOptions.js";
import { AnalysisInputError } from "../domain/analysisErrorCore.js";
/** Register provider-neutral binary overview and procedure CLI commands. */
export const registerCoreBinaryCommands = (
@@ -51,6 +52,7 @@ const registerOverviewCommands = (
}),
options: overviewOptions.extend({
integrityPolicy: javascriptApplicationOptions.shape.integrityPolicy,
sourcePath: javascriptApplicationOptions.shape.sourcePath,
}),
run: async ({ args, options }) => {
// Route JavaScript targets exactly like analyze-javascript-application,
@@ -63,19 +65,35 @@ const registerOverviewCommands = (
{
input_path: resolve(args.path),
integrity_policy: options.integrityPolicy,
...(options.sourcePath === undefined
? {}
: { source_paths: options.sourcePath }),
},
signal,
),
),
);
return logCliCommand(logger, "analyze", () =>
runDirectAnalysis(
return logCliCommand(logger, "analyze", () => {
if (options.sourcePath !== undefined)
throw new AnalysisInputError(
"analyze_javascript_application",
undefined,
[
{
path: ["source_paths"],
reason: "invalid_value",
message:
"Source selection requires the static JavaScript ASAR/directory analysis lane.",
},
],
);
return runDirectAnalysis(
args.path,
"binary_overview",
{},
directAnalysisOptionsFromCli(logger, options),
),
);
);
});
},
});
cli.command(CLI_COMMANDS.inspect, {
+3
View File
@@ -200,6 +200,9 @@ const registerJavaScriptApplicationCommand = (
input_path: args.path,
format: options.artifactFormat,
integrity_policy: options.integrityPolicy,
...(options.sourcePath === undefined
? {}
: { source_paths: options.sourcePath }),
...(options.maxHeapMb === undefined
? {}
: { max_heap_mb: options.maxHeapMb }),
Loaded 100 of 345 files, more files were not shown because too many files have changed in this diff. Show more