perf(ci): pack each test tier from its own timing history (#5799)

* perf(ci): pack each test tier from its own timing history

Fast shards were still ordered with the latest full-suite timing artifact,
which stayed stale across compiled-CLI session reuse. Publish and download
test-timings-fast and test-timings-full separately, and keep the legacy
artifact as a fallback until a tier has its own history.

* perf(ci): drop the shared timing-history fallback

Shard lookup now reads only the artifact for the current tier. A missing
artifact leaves history unset instead of packing from the old test-timings
file.

* test(ci): look up fast history when only full timings exist

The empty-history case now requests the fast tier against a full-tier
artifact and checks that the fast lookup ran and returned nothing. A
missing tier stays a separate test that does not query.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

.github/workflows/ci.yml

@@ -409,7 +409,9 @@ jobs:
         with:
           save: ${{ github.event_name != 'pull_request' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
       - run: npm run test:prepare
-      - name: Find available trusted full-suite timing history
+      # Fast and full shards measure different file costs. Download only the
+      # matching history so one tier cannot pack the other with its weights.
+      - name: Find available trusted timing history
         id: history
         continue-on-error: true
         env:
@@ -417,13 +419,14 @@ jobs:
           CI_REPOSITORY: ${{ github.repository }}
           CI_REPOSITORY_ID: ${{ github.repository_id }}
           CI_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+          CI_TIMING_TIER: ${{ needs.changes.outputs.tier }}
         run: node scripts/ci/test-history.mjs
       - name: Download previous timing report
         if: steps.history.outputs.run-id != ''
         continue-on-error: true
         uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
         with:
-          name: test-timings
+          name: ${{ steps.history.outputs.artifact-name }}
           run-id: ${{ steps.history.outputs.run-id }}
           github-token: ${{ github.token }}
           path: .cache/ci-test-history
@@ -555,11 +558,22 @@ jobs:
           include-hidden-files: true
           if-no-files-found: error
           retention-days: 30
+      # Default-branch runs only. Pull-request artifacts share this name and
+      # the lookup reads one page, so publishing them would hide main.
       - name: Publish successful timing history
-        if: needs.changes.outputs.code == 'true' && needs.changes.outputs.tier == 'full' && success() && needs.test-shard.result == 'success'
+        if: >-
+          success() &&
+          needs.changes.outputs.code == 'true' &&
+          needs.test-shard.result == 'success' &&
+          (
+            needs.changes.outputs.tier == 'full' ||
+            (needs.changes.outputs.tier == 'fast' && needs.test-source.result == 'success')
+          ) &&
+          github.ref == format('refs/heads/{0}', github.event.repository.default_branch) &&
+          (github.event_name == 'push' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
-          name: test-timings
+          name: test-timings-${{ needs.changes.outputs.tier }}
           path: .cache/ci-test-history/test-results.json
           include-hidden-files: true
           if-no-files-found: error

docs/testing.md

@@ -521,9 +521,15 @@ reports even when a shard fails, uploads available diagnostics, and writes the
 slowest files to the workflow summary. Partial reports are marked diagnostic
 only. Failed, cancelled, or missing selected shards still fail the required gate,
 and aggregate coverage thresholds remain enforced in full qualification.
-Only successful complete full-tier merges publish the small `test-timings` artifact, with
-checkout-relative paths; the build job downloads that artifact for the sequencer
-instead of the coverage/JUnit bundle. Timing weights come from Vitest module
+Only a successful complete merge on the default branch publishes shard history,
+and only from a push, schedule, or manual run. Fast runs publish
+`test-timings-fast`. Full runs publish `test-timings-full`. Pull requests do not
+publish it: the lookup reads one unpaginated artifact page, and pull-request
+artifacts would crowd out the default branch. The build job downloads the
+artifact that matches its tier, so fast shards use durations measured without
+coverage and full shards keep coverage-weighted durations. A tier with no
+qualifying artifact leaves history unset. The download is that small
+checkout-relative timing file, not the coverage/JUnit bundle. Timing weights come from Vitest module
 diagnostics and include tests and hooks, worker preparation, environment,
 imports, and setup. Import timings already include transform waits, which are
 not added again. The weights describe file costs under the measured worker and
@@ -1390,11 +1396,22 @@ catalog, and managed-conformance metadata. Each shard still installs the locked
 dependencies and runs its assigned tests; full qualification enables coverage. It no longer
 regenerates the same metadata independently.
 
-The build job also finds an available `test-timings` artifact from a successful
-trusted default-branch scheduled or manual qualification and includes that same
-report in each shard's snapshot. The bounded lookup verifies the artifact's
-producer; newer schedules that reuse qualification without producing timings do
-not hide the last full run's artifact.
+The build job finds the newest trusted timing artifact for its tier
+(`test-timings-fast` or `test-timings-full`) from a successful default-branch
+push, schedule, or manual CI run, and includes that same report in each shard's
+snapshot. The bounded lookup verifies the producer. Pull-request runs and other
+branches are not producers. A tier with no qualifying artifact leaves history
+unset, and Vitest uses its own assignment. A later schedule that reuses
+qualification without uploading timings leaves the previous artifact in place.
+
+On 2026-10-21 the newest full history was run
+`39545047294`, recorded before compiled-CLI session reuse. Fast run
+`39552964873` packed its eight built shards from that history, and those shard
+jobs executed for 319–591 seconds. Replaying that run's file durations through
+the sequencer's two-worker schedule balances every shard at about 404 seconds
+of file time. The same replay with the stale history weights leaves the slowest
+shard near 547 seconds and the fastest near 293. That spread is file-schedule
+time, not runner queue, which is why each tier keeps its own history.
 CI assigns discovered files by historical duration, with a deterministic
 fallback weight for new files, rather than dividing only by file count.
 Each file keeps its full duration on one worker. Extra workers overlap other

scripts/ci/test-history.mjs

@@ -4,21 +4,38 @@ import { resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { normalizeWorkflowRun } from "./workflow-run.mjs";
 
-const ARTIFACT_NAME = "test-timings";
+const TIER_EVENTS = ["push", "schedule", "workflow_dispatch"];
+const TIER_PROFILES = {
+  fast: { artifactName: "test-timings-fast", events: TIER_EVENTS },
+  full: { artifactName: "test-timings-full", events: TIER_EVENTS },
+};
 const MAX_AGE_MS = 30 * 24 * 60 * 60 * 1000;
 const MAX_CANDIDATES = 5;
+const EMPTY_SELECTION = { runId: "", artifactName: "" };
 const positiveId = (value) =>
   (typeof value === "string" || typeof value === "number") &&
   /^[1-9]\d*$/.test(String(value)) &&
   Number.isSafeInteger(Number(value));
 
-function findTimingRun({ repository, branch, repositoryId, now = Date.now() }) {
+function findTimingRun({
+  repository,
+  branch,
+  repositoryId,
+  tier,
+  now = Date.now(),
+}) {
   if (
     !/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repository ?? "") ||
     !branch ||
     !positiveId(repositoryId)
   )
-    return "";
+    return EMPTY_SELECTION;
+  const profile = Object.hasOwn(TIER_PROFILES, tier)
+    ? TIER_PROFILES[tier]
+    : null;
+  // Reused and other tiers do not publish shard history. Missing history uses
+  // Vitest's own assignment rather than another tier's artifact.
+  if (!profile) return EMPTY_SELECTION;
   const deadline = Date.now() + 20_000;
   const api = (...arguments_) => {
     const remaining = deadline - Date.now();
@@ -33,83 +50,93 @@ function findTimingRun({ repository, branch, repositoryId, now = Date.now() }) {
       }),
     );
   };
-
-  const { artifacts } = api(
-    `repos/${repository}/actions/artifacts`,
-    "-f",
-    `name=${ARTIFACT_NAME}`,
-    "-F",
-    "per_page=100",
-  );
-  if (!Array.isArray(artifacts))
-    throw new Error("Missing timing artifact inventory");
-  const candidates = artifacts
-    .filter((artifact) => {
-      const lineage = artifact?.workflow_run;
-      const age = now - Date.parse(artifact?.created_at);
-      return (
-        artifact?.name === ARTIFACT_NAME &&
-        artifact?.expired === false &&
-        typeof artifact?.created_at === "string" &&
-        Number.isFinite(age) &&
-        age >= 0 &&
-        age < MAX_AGE_MS &&
-        positiveId(lineage?.id) &&
-        positiveId(lineage?.repository_id) &&
-        positiveId(lineage?.head_repository_id) &&
-        lineage.repository_id === lineage.head_repository_id &&
-        lineage.repository_id === Number(repositoryId) &&
-        lineage.head_branch === branch
-      );
-    })
-    .sort(
-      (left, right) =>
-        Date.parse(right.created_at) - Date.parse(left.created_at),
-    )
-    .slice(0, MAX_CANDIDATES);
-
-  for (const artifact of candidates) {
-    const run = normalizeWorkflowRun(
-      api(`repos/${repository}/actions/runs/${artifact.workflow_run.id}`),
+  const search = ({ artifactName, events }) => {
+    const { artifacts } = api(
+      `repos/${repository}/actions/artifacts`,
+      "-f",
+      `name=${artifactName}`,
+      "-F",
+      "per_page=100",
     );
-    const runAge = now - Date.parse(run?.created_at);
-    if (
-      typeof run?.created_at === "string" &&
-      Number.isFinite(runAge) &&
-      runAge >= 0 &&
-      runAge < MAX_AGE_MS &&
-      run.id === artifact.workflow_run.id &&
-      run.status === "completed" &&
-      run.conclusion === "success" &&
-      run.workflowPath?.file === ".github/workflows/ci.yml" &&
-      ["schedule", "workflow_dispatch"].includes(run.event) &&
-      run.head_branch === branch &&
-      run.head_repository?.full_name === repository
-    )
-      return String(run.id);
-  }
-  return "";
+    if (!Array.isArray(artifacts))
+      throw new Error("Missing timing artifact inventory");
+    const candidates = artifacts
+      .filter((artifact) => {
+        const lineage = artifact?.workflow_run;
+        const age = now - Date.parse(artifact?.created_at);
+        return (
+          artifact?.name === artifactName &&
+          artifact?.expired === false &&
+          typeof artifact?.created_at === "string" &&
+          Number.isFinite(age) &&
+          age >= 0 &&
+          age < MAX_AGE_MS &&
+          positiveId(lineage?.id) &&
+          positiveId(lineage?.repository_id) &&
+          positiveId(lineage?.head_repository_id) &&
+          lineage.repository_id === lineage.head_repository_id &&
+          lineage.repository_id === Number(repositoryId) &&
+          lineage.head_branch === branch
+        );
+      })
+      .sort(
+        (left, right) =>
+          Date.parse(right.created_at) - Date.parse(left.created_at),
+      )
+      .slice(0, MAX_CANDIDATES);
+
+    for (const artifact of candidates) {
+      const run = normalizeWorkflowRun(
+        api(`repos/${repository}/actions/runs/${artifact.workflow_run.id}`),
+      );
+      const runAge = now - Date.parse(run?.created_at);
+      if (
+        typeof run?.created_at === "string" &&
+        Number.isFinite(runAge) &&
+        runAge >= 0 &&
+        runAge < MAX_AGE_MS &&
+        run.id === artifact.workflow_run.id &&
+        run.status === "completed" &&
+        run.conclusion === "success" &&
+        run.workflowPath?.file === ".github/workflows/ci.yml" &&
+        events.includes(run.event) &&
+        run.head_branch === branch &&
+        run.head_repository?.full_name === repository
+      )
+        return String(run.id);
+    }
+    return "";
+  };
+
+  const runId = search(profile);
+  return runId
+    ? { runId, artifactName: profile.artifactName }
+    : EMPTY_SELECTION;
 }
 
-function write(runId) {
+function write({ runId, artifactName }) {
   if (process.env.GITHUB_OUTPUT)
-    appendFileSync(process.env.GITHUB_OUTPUT, `run-id=${runId}\n`);
-  console.log(JSON.stringify({ run_id: runId }));
+    appendFileSync(
+      process.env.GITHUB_OUTPUT,
+      `run-id=${runId}\nartifact-name=${artifactName}\n`,
+    );
+  console.log(JSON.stringify({ run_id: runId, artifact_name: artifactName }));
 }
 
 if (
   process.argv[1] &&
   fileURLToPath(import.meta.url) === resolve(process.argv[1])
 ) {
-  let runId = "";
+  let selection = EMPTY_SELECTION;
   try {
-    runId = findTimingRun({
+    selection = findTimingRun({
       repository: process.env.CI_REPOSITORY,
       branch: process.env.CI_DEFAULT_BRANCH,
       repositoryId: process.env.CI_REPOSITORY_ID,
+      tier: process.env.CI_TIMING_TIER,
     });
   } catch {
     // History improves shard ordering but must never block CI.
   }
-  write(runId);
+  write(selection);
 }

tests/boundary/filesystem/ciTestHistory.test.ts

@@ -17,7 +17,7 @@ const now = Date.now();
 function artifact(id: number, overrides: Record<string, unknown> = {}) {
   return {
     id: id + 100,
-    name: "test-timings",
+    name: "test-timings-full",
     expired: false,
     created_at: new Date(now - 60_000).toISOString(),
     workflow_run: {
@@ -51,6 +51,7 @@ async function invoke(
     apiError?: boolean;
     repository?: string;
     repositoryId?: string;
+    tier?: string;
   } = {},
 ) {
   const directory = await createTestTempDirectory("rea-ci-test-history-");
@@ -92,6 +93,7 @@ async function invoke(
       CI_REPOSITORY: options.repository ?? repository,
       CI_REPOSITORY_ID: options.repositoryId ?? String(repositoryId),
       CI_DEFAULT_BRANCH: "main",
+      CI_TIMING_TIER: options.tier ?? "",
       GITHUB_OUTPUT: output,
     },
   });
@@ -120,17 +122,21 @@ it("selects the latest trusted full run that actually published timing history",
       14: run(14, { path: ".github/workflows/release.yml" }),
       11: run(11, { event: "workflow_dispatch" }),
     },
+    { tier: "full" },
   );
 
-  expect(JSON.parse(result.stdout)).toEqual({ run_id: "11" });
-  expect(output).toBe("run-id=11\n");
-  expect(calls).toContain("name=test-timings");
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "11",
+    artifact_name: "test-timings-full",
+  });
+  expect(output).toBe("run-id=11\nartifact-name=test-timings-full\n");
+  expect(calls).toContain("name=test-timings-full");
   expect(calls).toContain("per_page=100");
   expect(calls.match(/actions\/runs\//gu)).toHaveLength(3);
   expect(calls).not.toContain("actions/workflows");
 });
 
-it("rejects expired, foreign, old, and non-full run candidates", async () => {
+it("rejects expired, foreign, old, and pull-request candidates", async () => {
   const { result, output } = await invoke(
     [
       artifact(15, { expired: true }),
@@ -145,16 +151,30 @@ it("rejects expired, foreign, old, and non-full run candidates", async () => {
     {
       12: run(12, { event: "pull_request" }),
     },
+    { tier: "full" },
   );
 
-  expect(JSON.parse(result.stdout)).toEqual({ run_id: "" });
-  expect(output).toBe("run-id=\n");
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(output).toBe("run-id=\nartifact-name=\n");
 });
 
 it("writes an empty run id when GitHub history is unavailable", async () => {
-  const { result, output } = await invoke([], {}, { apiError: true });
-  expect(JSON.parse(result.stdout)).toEqual({ run_id: "" });
-  expect(output).toBe("run-id=\n");
+  const { result, output } = await invoke(
+    [],
+    {},
+    {
+      apiError: true,
+      tier: "full",
+    },
+  );
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(output).toBe("run-id=\nartifact-name=\n");
 });
 
 it("bounds run lookups to five artifacts even when inventory has more", async () => {
@@ -167,9 +187,13 @@ it("bounds run lookups to five artifacts even when inventory has more", async ()
         run(id, index < 5 ? { conclusion: "failure" } : {}),
       ]),
     ),
+    { tier: "full" },
   );
 
-  expect(JSON.parse(result.stdout)).toEqual({ run_id: "" });
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
   expect(calls.match(/actions\/runs\//gu)).toHaveLength(5);
 });
 
@@ -182,7 +206,119 @@ it("returns empty history without querying for an invalid repository identity",
       repositoryId: "not-an-id",
     },
   );
-  expect(JSON.parse(result.stdout)).toEqual({ run_id: "" });
-  expect(output).toBe("run-id=\n");
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(output).toBe("run-id=\nartifact-name=\n");
   expect(calls).toBe("");
 });
+
+it("selects fast history without reading the full-tier artifact", async () => {
+  const { result, calls } = await invoke(
+    [
+      artifact(32, {
+        created_at: new Date(now - 30_000).toISOString(),
+      }),
+      artifact(31, {
+        name: "test-timings-fast",
+        created_at: new Date(now - 90_000).toISOString(),
+      }),
+      artifact(30, {
+        name: "test-timings-fast",
+        created_at: new Date(now - 20_000).toISOString(),
+        workflow_run: {
+          ...artifact(30).workflow_run,
+          head_branch: "feature",
+        },
+      }),
+    ],
+    {
+      32: run(32, { event: "schedule" }),
+      31: run(31, { event: "push" }),
+    },
+    { tier: "fast" },
+  );
+
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "31",
+    artifact_name: "test-timings-fast",
+  });
+  expect(calls).toContain("name=test-timings-fast");
+  expect(calls).not.toContain("name=test-timings-full");
+  expect(calls.match(/actions\/runs\//gu)).toHaveLength(1);
+});
+
+it("ignores pull-request fast history instead of borrowing another artifact", async () => {
+  const { result, calls } = await invoke(
+    [artifact(41, { name: "test-timings-fast" }), artifact(40)],
+    {
+      41: run(41, { event: "pull_request" }),
+      40: run(40, { event: "workflow_dispatch" }),
+    },
+    { tier: "fast" },
+  );
+
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(calls).toContain("name=test-timings-fast");
+  expect(calls).not.toContain("name=test-timings-full");
+  expect(calls.match(/actions\/runs\//gu)).toHaveLength(1);
+});
+
+it("leaves fast history empty when only full-tier history was published", async () => {
+  const { result, output, calls } = await invoke(
+    [artifact(70)],
+    { 70: run(70, { event: "schedule" }) },
+    { tier: "fast" },
+  );
+
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(output).toBe("run-id=\nartifact-name=\n");
+  expect(calls).toContain("name=test-timings-fast");
+  expect(calls).not.toContain("name=test-timings-full");
+  expect(calls).not.toMatch(/actions\/runs\//u);
+});
+
+it("does not query when the tier has no timing artifact", async () => {
+  const { result, output, calls } = await invoke([artifact(71)], {
+    71: run(71, { event: "schedule" }),
+  });
+
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "",
+    artifact_name: "",
+  });
+  expect(output).toBe("run-id=\nartifact-name=\n");
+  expect(calls).toBe("");
+});
+
+it("selects full-tier history without using a newer fast artifact", async () => {
+  const { result } = await invoke(
+    [
+      artifact(61, {
+        name: "test-timings-fast",
+        created_at: new Date(now - 10_000).toISOString(),
+      }),
+      artifact(60, {
+        name: "test-timings-full",
+        created_at: new Date(now - 80_000).toISOString(),
+      }),
+    ],
+    {
+      61: run(61, { event: "push" }),
+      60: run(60, { event: "schedule" }),
+    },
+    { tier: "full" },
+  );
+
+  expect(JSON.parse(result.stdout)).toEqual({
+    run_id: "60",
+    artifact_name: "test-timings-full",
+  });
+});

tests/boundary/filesystem/ciTestReports.test.ts

@@ -243,10 +243,22 @@ it("keeps diagnosis available after merge failure and publishes only complete su
   const diagnosis = steps.find((step) => step.with?.name === "test-results");
   expect(diagnosis?.if).toContain("!cancelled()");
   expect(diagnosis?.if).not.toContain("success()");
-  const history = steps.find((step) => step.with?.name === "test-timings");
+  const history = steps.find(
+    (step) => step.name === "Publish successful timing history",
+  );
   expect(history?.if).toContain("success()");
   expect(history?.if).toContain("needs.test-shard.result == 'success'");
   expect(history?.if).toContain("needs.changes.outputs.tier == 'full'");
+  expect(history?.if).toContain(
+    "needs.changes.outputs.tier == 'fast' && needs.test-source.result == 'success'",
+  );
+  expect(history?.if).toContain("github.event_name == 'push'");
+  expect(history?.if).toContain(
+    "github.ref == format('refs/heads/{0}', github.event.repository.default_branch)",
+  );
+  expect(history?.with?.name).toBe(
+    "test-timings-${{ needs.changes.outputs.tier }}",
+  );
   expect(history?.with?.path).toBe(".cache/ci-test-history/test-results.json");
   const buildSteps = (await workflow()).build!.steps;
   const lookup = buildSteps.find((step) =>
@@ -256,11 +268,14 @@ it("keeps diagnosis available after merge failure and publishes only complete su
     CI_REPOSITORY: "${{ github.repository }}",
     CI_REPOSITORY_ID: "${{ github.repository_id }}",
     CI_DEFAULT_BRANCH: "${{ github.event.repository.default_branch }}",
+    CI_TIMING_TIER: "${{ needs.changes.outputs.tier }}",
   });
   const download = buildSteps.find(
     (step) => step.name === "Download previous timing report",
   );
-  expect(download?.with?.name).toBe(history?.with?.name);
+  expect(download?.with?.name).toBe(
+    "${{ steps.history.outputs.artifact-name }}",
+  );
   expect(download?.with?.["run-id"]).toBe(
     "${{ steps.history.outputs.run-id }}",
   );