bench: fix the Modal card choice and record the first Modal runs

Modal checks the card a function is declared with when the app starts,
before with_options(gpu=...) applies. A function declared on an L4 and
called on a T4 therefore still failed wherever an L4 could not be had,
which made the workflow's card choice useless. The card is now read from
BENCH_GPU when modal_app.py declares the function, and the workflow sets
that variable instead of passing --gpu.

First runs on a Modal L4 (manifoldbt 0.28.1, CPU side on its 4 reserved
cores): 9.1x, 10.3x and 18.8x on the CPU job's three grids, every cell
bit-identical to the CPU, same data digests as the published CPU run,
two minutes from start to finish. The README now quotes them.

With five repetitions a single slow one (48 ms among 32 ms) flagged the
first point as noisy; seven, as in the CPU benchmark, did not, so seven
is the default. The container saw exactly the 4 reserved cores. Its CPU
model is not readable inside Modal's sandbox: cpu_model now also tries
lscpu and the vendor, family and model fields before recording unknown.

.github/workflows/bench-gpu.yml

@@ -33,7 +33,7 @@ on:
       reps:
         description: "Interleaved repetitions per side and point"
         required: false
-        default: "5"
+        default: "7"
   release:
     types: [published]
   schedule:
@@ -61,7 +61,8 @@ jobs:
       measured: ${{ steps.measured.outputs.measured }}
     env:
       PYTHONUNBUFFERED: "1"
-      GPU: ${{ inputs.gpu || 'L4' }}
+      # Read by modal_app.py when it declares the function.
+      BENCH_GPU: ${{ inputs.gpu || 'L4' }}
     steps:
       - uses: actions/checkout@v4
 
@@ -117,7 +118,7 @@ jobs:
             exit 1
           fi
           echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
-          echo "benchmarking manifoldbt ${VERSION} on a Modal ${GPU}"
+          echo "benchmarking manifoldbt ${VERSION} on a Modal ${BENCH_GPU}"
 
       - name: Install the Modal client
         if: steps.token.outputs.present == 'true'
@@ -137,8 +138,7 @@ jobs:
           MANIFOLDBT_VERSION: ${{ steps.version.outputs.version }}
         run: |
           modal run modal_app.py \
-            --gpu "$GPU" \
-            --reps "${{ inputs.reps || '5' }}" \
+            --reps "${{ inputs.reps || '7' }}" \
             --sweep "${{ inputs.sweep }}" \
             --out results-gpu.json
 

benchmarks/gpu/README.md

@@ -28,10 +28,10 @@ and publishes it.
 Inside the container nothing differs from a user's machine: the published wheel
 from PyPI with its `[gpu]` extra, the libraries pinned in `requirements-lock.txt`,
 and an NVIDIA L4. Modal bills by the second for the life of the container, card and
-reserved cores included, so a run costs its wall time. By the CPU job's timings
-the three default points should take about a minute of measurement, most of it
-the CPU side, and the function's 15-minute timeout is the ceiling on what a stuck
-run can cost.
+reserved cores included, so a run costs its wall time: two minutes from start to
+finish on the first runs, image already built, most of it the CPU side of the
+comparison. The function's 15-minute timeout is the ceiling on what a stuck run
+can cost.
 
 ## What the harness refuses to publish
 
@@ -58,15 +58,20 @@ run can cost.
 2. Per point, one untimed call on each side. It compiles the kernels on the GPU,
    which is why it is reported on its own (`first_call_s`): it is what a user's
    first sweep costs, not what a sweep costs.
-3. `--reps` timed repetitions per side (5 by default), GPU and CPU interleaved, in
-   an order that alternates, so a clock that drifts over the run drifts on both
-   sides. Medians are published with min, max and IQR; a point whose IQR exceeds
-   15% of its median is flagged, as in the CPU benchmark.
+3. `--reps` timed repetitions per side (7 by default, as in the CPU benchmark),
+   GPU and CPU interleaved, in an order that alternates, so a clock that drifts
+   over the run drifts on both sides. Medians are published with min, max and
+   IQR; a point whose IQR exceeds 15% of its median is flagged, as in the CPU
+   benchmark. Seven rather than five because one slow repetition out of five is
+   enough to cross that line: it did on the first Modal run.
 
 The CPU side runs on 4 cores reserved for it as a hard limit, like the four vCPUs
 of the GitHub runner, and is told so (`max_parallelism`). A container can see
 more cores than it may use, and a sweep that spawned one thread per visible core
-on a 4-core quota would publish the cost of the oversubscription.
+on a 4-core quota would publish the cost of the oversubscription. What the
+container saw is recorded under `environment.cores`; on Modal it is the 4 cores
+reserved. The CPU model is not readable from inside Modal's sandbox, so it is
+recorded as unknown.
 
 ## Reading the numbers
 
@@ -74,13 +79,18 @@ The speedup is the GPU against the CPU of the **same machine**. It is not a rati
 against the CPU job, which runs on another machine: the two files are read side by
 side, never divided into each other.
 
-The ratio grows with the work per sweep. On a shared workstation (RTX 3090, CPU
-held to 4 threads, every point flagged noisy, so indicative only), the three
-default points ran roughly 5 to 9 times faster on the card, and a million bars by
-20,000 combinations about 20 times. That last point is not in the default set
-because its CPU side alone costs 37 s a call there, so a run would be minutes of
-reserved card mostly waiting for the CPU. It can be asked for on a manual run
-(`sweep: 1000000:20000`).
+The ratio grows with the work per sweep. On the first Modal runs (manifoldbt
+0.28.1, L4 against its 4 reserved cores), the three default points ran 9.1, 10.3
+and 18.8 times faster on the card: 33 ms against 296 ms, 111 ms against 1.15 s,
+and 233 ms against 4.39 s. The published file is the reference; these are what
+the harness was validated on.
+
+Larger grids widen the gap. On a shared workstation (RTX 3090, CPU held to 4
+threads, flagged noisy, so indicative only), a million bars by 20,000
+combinations ran about 20 times faster on the card. That point is not in the
+default set because its CPU side alone costs 37 s a call there, so a run would be
+minutes of reserved card mostly waiting for the CPU. It can be asked for on a
+manual run (`sweep: 1000000:20000`).
 
 The numbers depend on the card. The published file is always an L4; a manual run
 can name another one, and is then reported without being published.
@@ -100,8 +110,8 @@ On Modal, as the workflow does:
 ```bash
 pip install -r requirements-ci.txt
 modal token new
-MANIFOLDBT_VERSION=0.28.1 MANIFOLDBT_CI_LICENSE=<key> \
-    modal run modal_app.py --gpu L4 --out results-gpu.json
+BENCH_GPU=L4 MANIFOLDBT_VERSION=0.28.1 MANIFOLDBT_CI_LICENSE=<key> \
+    modal run modal_app.py --out results-gpu.json
 ```
 
 In a fork, the workflow needs three repository secrets: `MODAL_TOKEN_ID` and

benchmarks/gpu/bench.py

@@ -6,7 +6,7 @@ measure the CPU path. This one runs the *same* sweep -- same generator, same see
 same grid, same strategy, imported from ``../vs_vectorbt`` rather than copied --
 on a GPU, and on the CPU of the same machine beside it.
 
-    python bench.py --sweep 20000:5000 20000:20000 200000:10000 --reps 5 \\
+    python bench.py --sweep 20000:5000 20000:20000 200000:10000 --reps 7 \\
         --out results-gpu.json
 
 It runs on any machine with an NVIDIA card, its driver, ``manifoldbt[gpu]`` and a
@@ -109,13 +109,35 @@ def cgroup_quota(path: str = "/sys/fs/cgroup/cpu.max") -> Optional[float]:
 
 
 def cpu_model() -> str:
+    """The CPU's name, from whichever source this kernel fills in.
+
+    A sandboxed kernel (Modal runs gVisor) can leave ``model name`` out of
+    /proc/cpuinfo, so lscpu and the vendor/family/model numbers are tried next.
+    """
+    info: Dict[str, str] = {}
     try:
         with open("/proc/cpuinfo", encoding="utf-8") as fh:
             for line in fh:
-                if line.startswith("model name"):
-                    return line.split(":", 1)[1].strip()
+                key, sep, value = line.partition(":")
+                if sep and key.strip() not in info:
+                    info[key.strip()] = value.strip()
     except OSError:
         pass
+    if info.get("model name"):
+        return info["model name"]
+    exe = shutil.which("lscpu")
+    if exe:
+        try:
+            out = subprocess.run([exe], capture_output=True, text=True, timeout=30).stdout
+            for line in out.splitlines():
+                key, _, value = line.partition(":")
+                if key.strip() == "Model name" and value.strip():
+                    return value.strip()
+        except (OSError, subprocess.SubprocessError):
+            pass
+    if info.get("vendor_id"):
+        return "{} family {} model {}".format(
+            info["vendor_id"], info.get("cpu family", "?"), info.get("model", "?"))
     return platform.processor() or "unknown"
 
 
@@ -315,8 +337,8 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
     parser.add_argument("--sweep", nargs="+", type=parse_point,
                         default=[parse_point(p) for p in DEFAULT_POINTS],
                         metavar="BARS:COMBOS", help="grid points (default: the CPU job's)")
-    parser.add_argument("--reps", type=int, default=5,
-                        help="timed repetitions per side and point (default 5)")
+    parser.add_argument("--reps", type=int, default=7,
+                        help="timed repetitions per side and point (default 7)")
     parser.add_argument("--cpu-threads", type=int, default=0,
                         help="threads for the CPU side (default: the usable cores)")
     parser.add_argument("--platform", default="",

benchmarks/gpu/modal_app.py

@@ -1,7 +1,7 @@
 """Run bench.py on a Modal GPU, from the workflow or from a laptop.
 
-    MANIFOLDBT_VERSION=0.28.1 MANIFOLDBT_CI_LICENSE=<key> \\
-        modal run modal_app.py --gpu L4 --reps 5 --out results-gpu.json
+    BENCH_GPU=L4 MANIFOLDBT_VERSION=0.28.1 MANIFOLDBT_CI_LICENSE=<key> \\
+        modal run modal_app.py --out results-gpu.json
 
 GitHub's hosted runners have no GPU, and its GPU runners are billed by the minute
 on paid plans only, so the card comes from Modal. Modal bills by the second for
@@ -40,7 +40,11 @@ MEMORY_MIB = 8192
 # The whole run, image pull included, measured at a few minutes. The ceiling on
 # what a stuck run can cost.
 TIMEOUT_S = 15 * 60
-DEFAULT_GPU = "L4"
+# The card is read when the function is declared, not passed at call time: Modal
+# checks the declared card when the app starts, so a function declared on an L4
+# and called `with_options(gpu="T4")` still fails wherever an L4 cannot be had.
+# Passed into the container too, so both sides declare the same function.
+GPU = os.environ.get("BENCH_GPU", "").strip() or "L4"
 
 VERSION = os.environ.get("MANIFOLDBT_VERSION", "").strip()
 if modal.is_local() and not VERSION:
@@ -68,7 +72,8 @@ app = modal.App("manifoldbt-gpu-bench")
 
 @app.function(
     image=image,
-    gpu=DEFAULT_GPU,
+    gpu=GPU,
+    env={"BENCH_GPU": GPU},
     cpu=(CPU_CORES, CPU_CORES),
     memory=MEMORY_MIB,
     timeout=TIMEOUT_S,
@@ -94,16 +99,15 @@ def measure(argv: list) -> dict:
 
 
 @app.local_entrypoint()
-def main(gpu: str = DEFAULT_GPU, reps: int = 5, sweep: str = "",
-         out: str = "results-gpu.json") -> None:
+def main(reps: int = 7, sweep: str = "", out: str = "results-gpu.json") -> None:
     argv = [
         "--reps", str(reps),
         "--cpu-threads", str(CPU_CORES),
-        "--platform", "Modal {}, CPU side on {} reserved cores".format(gpu, CPU_CORES),
+        "--platform", "Modal {}, CPU side on {} reserved cores".format(GPU, CPU_CORES),
     ]
     if sweep.strip():
         argv += ["--sweep", *sweep.split()]
-    reply = measure.with_options(gpu=gpu).remote(argv)
+    reply = measure.remote(argv)
     if reply["result"] is not None:
         pathlib.Path(out).write_text(reply["result"], encoding="utf-8")
         print("wrote " + out)