gavindiaz/manifoldbt
1
0
Fork 0
mirror of https://github.com/manifoldbt/manifoldbt.git synced 2026-11-02 07:56:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench: time only the GPU on Modal, against the CPU job's published numbers

Browse Source 
The CPU reference for a sweep is the CPU job's, published from its
GitHub runner. Timing the CPU again in the Modal container added a second
CPU number from another machine, most of the run's billed time, and a
ratio that measured the container's CPU as much as the card.

The CPU path now runs once per point, untimed, only so that every GPU
cell is still checked against it. The report reads the CPU column from
vs_vectorbt/results/latest.json, matched on bars, combinations and data
digest, and gives a ratio only when both runs measured the same version.

Validated on a Modal L4 with manifoldbt 0.28.1: 32 ms, 107 ms and 223 ms
against 228 ms, 902 ms and 3.53 s in the CPU job's run of the same
version (7.1x, 8.4x, 15.9x), every cell bit-identical, 32 s from start to
finish instead of 120 s. A local run on a different numpy produced other
data digests and was, as intended, not matched with the CPU job.
This commit is contained in:
Jimmy7892
2026-10-25 11:25:42 +01:00
parent 1fb3b58944
commit 923559d68a
6 changed files with 206 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/gpu/README.md
+50 -49
View File
@@ -2,14 +2,14 @@
The sweeps job of the [CPU benchmark](../vs_vectorbt/README.md#parameter-sweeps)
runs on a GitHub-hosted runner, which has no GPU, so it can only ever measure the
CPU path of a sweep. This benchmark runs the same sweeps on a GPU, and on the CPU
of the same machine beside it.
CPU path of a sweep. This benchmark times the same sweeps on a GPU. The CPU number
they are read against stays the CPU job's, published from its runner.
Same sweeps means the same code, not a copy of it: the data generator, the seed,
the grid and the strategy are imported from `../vs_vectorbt` (`data.py`,
`sweep_child.py`), and the default points are the CPU job's three. Each point
records the digest of its data, so its GPU number and the CPU job's number for
the same grid can be checked to be about the same bars.
records the digest of its data, and the report matches a GPU point with a CPU
point only when bars, combinations and digest all agree.
| bars | combinations | grid (fast x slow) |
|---:|---:|---|
@@ -28,10 +28,9 @@ and publishes it.
Inside the container nothing differs from a user's machine: the published wheel
from PyPI with its `[gpu]` extra, the libraries pinned in `requirements-lock.txt`,
and an NVIDIA L4. Modal bills by the second for the life of the container, card and
reserved cores included, so a run costs its wall time: two minutes from start to
finish on the first runs, image already built, most of it the CPU side of the
comparison. The function's 15-minute timeout is the ceiling on what a stuck run
can cost.
reserved cores included, so a run costs its wall time: about 30 seconds from
start to finish, image already built. The function's 15-minute timeout is the
ceiling on what a stuck run can cost.
## What the harness refuses to publish
@@ -40,12 +39,13 @@ can cost.
take the sweep, so that a user's results are never affected. Here, any
`UserWarning` raised during a GPU sweep fails the point. A machine where no card
is visible fails before the first point.
- **A timing for different work.** Every cell of the GPU grid is compared with the
CPU grid, keyed by its parameter pair, through the gate the CPU job applies to
vectorbt: total return and maximum drawdown, worst cell against 1e-9 of capital.
The engine documents the two paths as identical in fp64, so the result also
counts the cells that are identical to the bit. Every timed repetition must
reproduce the first call, cell for cell.
- **A timing for different work.** Each point also runs once on the CPU of the GPU
machine, untimed, and every cell of the GPU grid is compared with that CPU grid,
keyed by its parameter pair, through the gate the CPU job applies to vectorbt:
total return and maximum drawdown, worst cell against 1e-9 of capital. The
engine documents the two paths as identical in fp64, so the result also counts
the cells that are identical to the bit. Every timed repetition must reproduce
the first call, cell for cell.
- **A timing measured across a licence change**, as in the CPU job. GPU
acceleration is a Researcher feature, and the run activates the same CI licence
as the CPU sweeps job.
@@ -55,42 +55,41 @@ can cost.
1. `calibrate_gpu()`, untimed: a few seconds that bring the card to its steady
clocks and measure the costs the sweep uses to choose its execution path. The
measured constants are kept in the result.
2. Per point, one untimed call on each side. It compiles the kernels on the GPU,
which is why it is reported on its own (`first_call_s`): it is what a user's
first sweep costs, not what a sweep costs.
3. `--reps` timed repetitions per side (7 by default, as in the CPU benchmark),
GPU and CPU interleaved, in an order that alternates, so a clock that drifts
over the run drifts on both sides. Medians are published with min, max and
IQR; a point whose IQR exceeds 15% of its median is flagged, as in the CPU
benchmark. Seven rather than five because one slow repetition out of five is
enough to cross that line: it did on the first Modal run.
2. Per point, one untimed GPU call. It compiles the kernels, which is why it is
reported on its own (`first_call_s`): it is what a user's first sweep costs,
not what a sweep costs.
3. `--reps` timed GPU repetitions (7 by default, as in the CPU benchmark). Medians
are published with min, max and IQR; a point whose IQR exceeds 15% of its
median is flagged, as in the CPU benchmark. Seven rather than five because one
slow repetition out of five is enough to cross that line: it did on the first
Modal run.
The CPU side runs on 4 cores reserved for it as a hard limit, like the four vCPUs
of the GitHub runner, and is told so (`max_parallelism`). A container can see
more cores than it may use, and a sweep that spawned one thread per visible core
on a 4-core quota would publish the cost of the oversubscription. What the
container saw is recorded under `environment.cores`; on Modal it is the 4 cores
reserved. The CPU model is not readable from inside Modal's sandbox, so it is
recorded as unknown.
The CPU parity run gets 4 reserved cores, as a hard limit, and is told so
(`max_parallelism`): the card is billed while it runs, so it should be short, and
a sweep that spawned one thread per visible core on a smaller quota would crawl.
What the container saw is recorded under `environment.cores`; on Modal it is the
4 cores reserved. The CPU model is not readable from inside Modal's sandbox, so it
is recorded as unknown. None of this is a CPU timing: there is none in the result.
## Reading the numbers
The speedup is the GPU against the CPU of the **same machine**. It is not a ratio
against the CPU job, which runs on another machine: the two files are read side by
side, never divided into each other.
The report puts the GPU median next to the CPU job's published median for the same
grid, and divides one by the other only when both runs measured the same
manifoldbt version. The two come from different machines, an L4 on Modal and a
4-vCPU GitHub runner, and the ratio says exactly that: what the sweep costs on the
runner the CPU numbers are published from, against what it costs on the card. It
is not the speed of the card against the CPU next to it, which depends as much on
that CPU as on the card.
The ratio grows with the work per sweep. On the first Modal runs (manifoldbt
0.28.1, L4 against its 4 reserved cores), the three default points ran 9.1, 10.3
and 18.8 times faster on the card: 33 ms against 296 ms, 111 ms against 1.15 s,
and 233 ms against 4.39 s. The published file is the reference; these are what
the harness was validated on.
On the validation run (manifoldbt 0.28.1), the three default points took 32 ms,
107 ms and 223 ms on the L4, against 228 ms, 902 ms and 3.53 s in the CPU job's
published run of the same version: 7.1, 8.4 and 15.9 times faster on the card.
The published files are the reference; these are what the harness was validated
on.
Larger grids widen the gap. On a shared workstation (RTX 3090, CPU held to 4
threads, flagged noisy, so indicative only), a million bars by 20,000
combinations ran about 20 times faster on the card. That point is not in the
default set because its CPU side alone costs 37 s a call there, so a run would be
minutes of reserved card mostly waiting for the CPU. It can be asked for on a
manual run (`sweep: 1000000:20000`).
Larger grids widen the gap, and a manual run can ask for one
(`sweep: 1000000:20000`). The CPU job has no number for it, so the report shows the
GPU alone, and the untimed CPU parity run then costs about a minute of the run.
The numbers depend on the card. The published file is always an L4; a manual run
can name another one, and is then reported without being published.
@@ -114,15 +113,17 @@ BENCH_GPU=L4 MANIFOLDBT_VERSION=0.28.1 MANIFOLDBT_CI_LICENSE=<key> \
modal run modal_app.py --out results-gpu.json
```
In a fork, the workflow needs three repository secrets: `MODAL_TOKEN_ID` and
`MODAL_TOKEN_SECRET` (a Modal token) and `MANIFOLDBT_CI_LICENSE`. Without the Modal
token it measures nothing and says so in a warning instead of failing.
Modal runs GPU functions only on an account with a payment method. In a fork, the
workflow needs three repository secrets: `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`
(a Modal token) and `MANIFOLDBT_CI_LICENSE`. Without the Modal token it measures
nothing and says so in a warning instead of failing.
## Files
- `bench.py` - the harness: calibration, the gates, the timings, one JSON
- `bench.py` - the harness: calibration, the gates, the GPU timings, one JSON
- `modal_app.py` - runs `bench.py` on a Modal GPU and hands back its JSON
- `report.py` - JSON to Markdown, and to the GitHub job summary
- `report.py` - JSON to Markdown, with the CPU job's numbers beside it, and to the
GitHub job summary
- `publish.py` - wraps a green run with its provenance into `results/latest.json`
- `test_bench.py` - guards for the gates, the report and the publisher; no GPU and
no engine needed
benchmarks/gpu/bench.py
+39 -44
View File
@@ -2,9 +2,11 @@
The sweeps job of ``bench-vs-vectorbt.yml`` times manifoldbt's parameter sweep on
a four-vCPU GitHub runner. Those runners have no GPU, so that job can only ever
measure the CPU path. This one runs the *same* sweep -- same generator, same seed,
same grid, same strategy, imported from ``../vs_vectorbt`` rather than copied --
on a GPU, and on the CPU of the same machine beside it.
measure the CPU path. This one times the *same* sweep -- same generator, same
seed, same grid, same strategy, imported from ``../vs_vectorbt`` rather than
copied -- on a GPU. The CPU number it is read against is that job's, published
from the runner: the CPU path runs here once per point, untimed, only so that
every GPU cell can be checked against it.
python bench.py --sweep 20000:5000 20000:20000 200000:10000 --reps 7 \\
--out results-gpu.json
@@ -34,15 +36,11 @@ The card is calibrated first (``calibrate_gpu``, untimed, a few seconds: it
brings the card to its steady clocks and measures the costs the sweep uses to
pick its path), and every point is run once untimed before it is timed, so that
compiling kernels is not counted as sweep time. That first call is still
reported on its own, because it is what a user's first sweep costs. GPU and CPU
repetitions are interleaved, and their order alternates, so a clock that drifts
over the run drifts on both sides.
reported on its own, because it is what a user's first sweep costs.
The CPU side is given the cores the machine lets this process use, read from its
CPU affinity and cgroup quota rather than from ``os.cpu_count()``: a container
that sees 64 host cores and may use 4 of them would otherwise run 64 threads on
4 cores and publish the cost of the oversubscription. ``--cpu-threads`` sets it
outright, which is what ``modal_app.py`` does with the cores it reserved.
The parity run on the CPU is given the cores the machine lets this process use,
read from its CPU affinity and cgroup quota rather than from ``os.cpu_count()``,
or ``--cpu-threads``. It is not timed: the CPU number is the runner's.
"""
from __future__ import annotations
@@ -72,8 +70,8 @@ SCHEMA_VERSION = 1
# Same flag as the CPU benchmark: a point whose interquartile range exceeds 15%
# of its median is indicative only.
NOISE_THRESHOLD = 0.15
# The CPU sweeps job's three points, so that every GPU number has a CPU number
# for the same grid on the same data in the published CPU results.
# The CPU sweeps job's three points: the GPU number of each is read against the
# CPU number that job publishes for the same grid on the same data.
DEFAULT_POINTS = ("20000:5000", "20000:20000", "200000:10000")
@@ -257,41 +255,38 @@ def measure_point(bars: int, combos: int, reps: int, threads: int) -> Dict[str,
cpu = run_mbt(df, fast_vals, slow_vals, os.path.join(work, "cpu"),
metrics=True, device="cpu", max_parallelism=threads)
# The untimed first calls: kernels compile here, and the results they
# return are the ones every repetition must reproduce.
# The untimed first GPU call: kernels compile here, and the results it
# returns are the ones every repetition must reproduce.
gpu_first, gpu_ref, fallbacks = timed(gpu)
cpu_first, cpu_ref, _ = timed(cpu)
# The CPU path, once and untimed: the reference every GPU cell is
# checked against. Its timing is the CPU job's, not this machine's.
_, cpu_ref, _ = timed(cpu)
samples: Dict[str, List[float]] = {"gpu": [], "cpu": []}
unstable = {"gpu": 0, "cpu": 0}
for rep in range(reps):
order = (("gpu", gpu), ("cpu", cpu)) if rep % 2 == 0 else (("cpu", cpu), ("gpu", gpu))
for side, call in order:
elapsed, out, caught = timed(call)
samples[side].append(elapsed)
if side == "gpu":
fallbacks.extend(caught)
ref = gpu_ref if side == "gpu" else cpu_ref
if len(out) != len(ref) or identical_cells(ref, out) != len(ref):
unstable[side] += 1
samples: List[float] = []
unstable = 0
for _rep in range(reps):
elapsed, out, caught = timed(gpu)
samples.append(elapsed)
fallbacks.extend(caught)
if len(out) != len(gpu_ref) or identical_cells(gpu_ref, out) != len(gpu_ref):
unstable += 1
finally:
shutil.rmtree(work, ignore_errors=True)
parity = compare(cpu_ref, gpu_ref)
parity["reference"] = "cpu"
parity["reference_threads"] = threads
parity["identical_cells"] = identical_cells(cpu_ref, gpu_ref)
stats = {side: summarise(values) for side, values in samples.items()}
gpu_s, cpu_s = stats["gpu"]["median_s"], stats["cpu"]["median_s"]
stats = summarise(samples)
gpu_s = stats["median_s"]
entry.update({
"parity": parity,
"timings": {
"seconds": {"gpu": gpu_s, "cpu": cpu_s},
"speedup": cpu_s / gpu_s if gpu_s else None,
"per_combo_us": {"gpu": gpu_s / entry["combos"] * 1e6,
"cpu": cpu_s / entry["combos"] * 1e6},
"first_call_s": {"gpu": gpu_first, "cpu": cpu_first},
"seconds": gpu_s,
"per_combo_us": gpu_s / entry["combos"] * 1e6,
"first_call_s": gpu_first,
"stats": stats,
"noisy": any(s["iqr_over_median"] > NOISE_THRESHOLD for s in stats.values()),
"noisy": stats["iqr_over_median"] > NOISE_THRESHOLD,
"reps": reps,
},
"fallbacks": [{"category": c, "message": m} for c, m in fallbacks],
@@ -310,7 +305,7 @@ def verdict(entry: Dict[str, Any]) -> Tuple[str, Optional[str]]:
return "failed", "the licence tier changed during the point"
if entry["parity"]["status"] != "exact":
return "failed", "GPU and CPU results differ beyond the gate"
if any(entry["unstable_reps"].values()):
if entry["unstable_reps"]:
return "failed", "a repetition did not reproduce the first call"
return "exact", None
@@ -338,9 +333,10 @@ def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
default=[parse_point(p) for p in DEFAULT_POINTS],
metavar="BARS:COMBOS", help="grid points (default: the CPU job's)")
parser.add_argument("--reps", type=int, default=7,
help="timed repetitions per side and point (default 7)")
help="timed GPU repetitions per point (default 7)")
parser.add_argument("--cpu-threads", type=int, default=0,
help="threads for the CPU side (default: the usable cores)")
help="threads for the untimed CPU parity run "
"(default: the usable cores)")
parser.add_argument("--platform", default="",
help="where this ran, as the report should name it")
parser.add_argument("--out", required=True, help="results JSON")
@@ -374,7 +370,7 @@ def main(argv: Optional[List[str]] = None) -> int:
env = result["environment"]
card = env["gpus"][0]["name"] if env["gpus"] else "no GPU visible"
print(f"manifoldbt {env['versions']['manifoldbt']} | {card} | "
f"CPU {env['cpu']} x {threads} threads", flush=True)
f"CPU parity run on {threads} threads", flush=True)
if not tier()["pro"]:
result["reason"] = "no licensed tier: GPU sweeps are not run unlicensed"
@@ -399,10 +395,9 @@ def main(argv: Optional[List[str]] = None) -> int:
write(args.out, result) # a crash further on still leaves this point
if entry["status"] == "exact":
t = entry["timings"]
print("{:>9,} bars x {:>6,} combos | GPU {:8.3f} s | CPU {:8.3f} s | "
"x{:.1f} | first GPU call {:.3f} s | {}/{} cells bit-identical{}".format(
bars, entry["combos"], t["seconds"]["gpu"], t["seconds"]["cpu"],
t["speedup"], t["first_call_s"]["gpu"],
print("{:>9,} bars x {:>6,} combos | GPU {:8.3f} s | first call {:.3f} s | "
"{}/{} cells bit-identical to the CPU{}".format(
bars, entry["combos"], t["seconds"], t["first_call_s"],
entry["parity"]["identical_cells"], entry["combos"],
" | NOISY" if t["noisy"] else ""), flush=True)
else:
benchmarks/gpu/modal_app.py
+8 -6
View File
@@ -6,8 +6,10 @@
GitHub's hosted runners have no GPU, and its GPU runners are billed by the minute
on paid plans only, so the card comes from Modal. Modal bills by the second for
the life of the container, card and reserved cores included, and nothing while no
container runs, so a run costs its wall time: a few minutes of an L4, which is a
few cents. The function's timeout below is the ceiling on that cost.
container runs, so a run costs its wall time: about half a minute of an L4, a
cent or two. The function's timeout below is the ceiling on that cost. Only the
GPU is timed here; the CPU number the GPU is read against is the one the CPU
sweeps job publishes from its GitHub runner.
The image is the published wheel from PyPI with its ``[gpu]`` extra (the NVIDIA
runtime compiler that compiles the kernels; the driver comes with the card), plus
@@ -31,9 +33,9 @@ HERE = pathlib.Path(__file__).resolve().parent
BENCHMARKS = HERE.parent
REMOTE = "/root/benchmarks"
# Cores reserved for the CPU side of the comparison, as both the request and the
# hard limit, so the CPU number is not a measure of how idle the host was. Four,
# like the GitHub runner the CPU sweeps job runs on.
# Cores for the untimed CPU run every GPU cell is checked against, as both the
# request and the hard limit. The card is billed while that run goes, so it gets
# enough cores to be short; four of them cost a fraction of the card.
CPU_CORES = 4
# Room for the 1M-bar point; the sweeps themselves stay well under it.
MEMORY_MIB = 8192
@@ -103,7 +105,7 @@ def main(reps: int = 7, sweep: str = "", out: str = "results-gpu.json") -> None:
argv = [
"--reps", str(reps),
"--cpu-threads", str(CPU_CORES),
"--platform", "Modal {}, CPU side on {} reserved cores".format(GPU, CPU_CORES),
"--platform", "Modal {}".format(GPU),
]
if sweep.strip():
argv += ["--sweep", *sweep.split()]
benchmarks/gpu/report.py
+68 -34
View File
@@ -1,12 +1,14 @@
"""Render a GPU sweep result as Markdown, and into the GitHub job summary.
python report.py results-gpu.json
python report.py results-gpu.json [--cpu-results ../vs_vectorbt/results/latest.json]
The speedup in the table is GPU against the CPU of the same machine, on the cores
named in the header. It is not a comparison with the CPU sweeps job, which runs
on another machine: that job's numbers for the same grids are in
../vs_vectorbt/results/latest.json, and they are read side by side, not divided
by these.
The CPU column is not measured here. It is the CPU sweeps job's published number
for the same grid, read from its results file and matched on bars, combinations
and the digest of the data, so both columns are about the same work. The two
come from different machines (a GitHub runner and a Modal GPU), which is what
the ratio says: what the sweep costs on the runner the CPU numbers are published
from, against what it costs on the card. It is only computed when both runs
measured the same manifoldbt version.
"""
from __future__ import annotations
@@ -14,12 +16,33 @@ import argparse
import json
import os
import sys
from typing import Any, Dict, List
from typing import Any, Dict, List, Optional, Tuple
HERE = os.path.dirname(os.path.abspath(__file__))
CPU_RESULTS = os.path.join(HERE, "..", "vs_vectorbt", "results", "latest.json")
METHOD_LINK = (
"https://github.com/manifoldbt/manifoldbt/blob/master/benchmarks/gpu/README.md"
)
Key = Tuple[int, int, str]
def cpu_reference(published: Optional[Dict[str, Any]]) -> Dict[str, Any]:
"""The CPU sweeps job's points, keyed by (bars, combos, data digest)."""
payload = (published or {}).get("sweeps") or {}
env = payload.get("environment") or {}
points: Dict[Key, float] = {}
for row in payload.get("sweeps") or []:
seconds = ((row.get("timings") or {}).get("seconds") or {}).get("manifoldbt")
if seconds is not None:
points[(row["bars"], row["combos"], row.get("data_digest", ""))] = seconds
return {
"points": points,
"version": (env.get("versions") or {}).get("manifoldbt"),
"cores": env.get("logical_cores"),
"run_url": (published or {}).get("run_url", ""),
}
def _card(env: Dict[str, Any]) -> str:
gpus = env.get("gpus") or []
@@ -35,56 +58,61 @@ def _seconds(value: float) -> str:
return "{:.3f} s".format(value) if value >= 0.1 else "{:.1f} ms".format(value * 1e3)
def render(payload: Dict[str, Any]) -> str:
def render(payload: Dict[str, Any], cpu: Optional[Dict[str, Any]] = None) -> str:
cpu = cpu or cpu_reference(None)
env = payload.get("environment") or {}
versions = env.get("versions") or {}
lines: List[str] = [
"## GPU parameter sweeps",
"",
"manifoldbt {} on {}. CPU side: {}, {} threads. {}".format(
versions.get("manifoldbt", "?"), _card(env), env.get("cpu", "?"),
env.get("cpu_threads", "?"), env.get("platform", "")).rstrip(),
"",
]
version = (env.get("versions") or {}).get("manifoldbt", "?")
same_version = cpu["version"] == version
lines: List[str] = ["## GPU parameter sweeps", "",
"manifoldbt {} on {}. {}".format(
version, _card(env), env.get("platform", "")).rstrip()]
if cpu["points"]:
lines.append(
"CPU column: the CPU sweeps job's published run{}, manifoldbt {} on a "
"{}-vCPU GitHub runner.{}".format(
" ({})".format(cpu["run_url"]) if cpu["run_url"] else "",
cpu["version"] or "?", cpu["cores"] or "?",
"" if same_version else
" Not the version measured here, so no ratio is given."))
lines.append("")
if payload.get("status") != "exact" and payload.get("reason"):
lines += ["**Failed: {}**".format(payload["reason"]), ""]
rows = payload.get("sweeps") or []
if rows:
lines += [
"| bars | combos | GPU | CPU | GPU vs CPU | per combo (GPU) | first GPU call "
"| parity |",
"| bars | combos | GPU | per combo | first GPU call | CPU (runner) "
"| GPU vs runner CPU | parity |",
"|---:|---:|---:|---:|---:|---:|---:|:---|",
]
for row in rows:
if row.get("status") != "exact" and "timings" not in row:
if "timings" not in row:
lines.append("| {:,} | {:,} | | | | | | failed: {} |".format(
row["bars"], row.get("requested_combos", 0), row.get("reason", "")))
continue
t = row["timings"]
noisy = " ~" if t.get("noisy") else ""
cpu_s = cpu["points"].get((row["bars"], row["combos"], row.get("data_digest", "")))
ratio = "x{:.1f}".format(cpu_s / t["seconds"]) if cpu_s and same_version else ""
parity = row["parity"]
verdict = "{} ({:,}/{:,} bit-identical)".format(
row["status"], parity.get("identical_cells", 0), row["combos"])
if row.get("reason"):
verdict += ": " + row["reason"]
lines.append(
"| {:,} | {:,} | {}{} | {}{} | x{:.1f} | {:.2f} us | {} | {} |".format(
row["bars"], row["combos"],
_seconds(t["seconds"]["gpu"]), noisy,
_seconds(t["seconds"]["cpu"]), noisy,
t["speedup"], t["per_combo_us"]["gpu"],
_seconds(t["first_call_s"]["gpu"]), verdict))
lines.append("| {:,} | {:,} | {}{} | {:.2f} us | {} | {} | {} | {} |".format(
row["bars"], row["combos"], _seconds(t["seconds"]),
" ~" if t.get("noisy") else "", t["per_combo_us"],
_seconds(t["first_call_s"]), _seconds(cpu_s) if cpu_s else "",
ratio, verdict))
lines.append("")
lines += [
"---",
"",
"Medians of {} interleaved repetitions per side, after an untimed first call "
"(reported apart: it compiles the kernels) and a calibration of the card. "
"`~` = IQR above 15% of the median, indicative only. Parity: every cell of the "
"GPU grid against the CPU grid, keyed by parameter pair. Method: {}".format(
payload.get("reps", "?"), METHOD_LINK),
"GPU: medians of {} repetitions, after an untimed first call (reported apart: "
"it compiles the kernels) and a calibration of the card. `~` = IQR above 15% of "
"the median, indicative only. Parity: every cell of the GPU grid against the "
"same grid run once on the CPU of the GPU machine, keyed by parameter pair. "
"Method: {}".format(payload.get("reps", "?"), METHOD_LINK),
"",
]
return "\n".join(lines)
@@ -93,12 +121,18 @@ def render(payload: Dict[str, Any]) -> str:
def main() -> int:
parser = argparse.ArgumentParser(description="render a GPU sweep result")
parser.add_argument("results", nargs="?", default="results-gpu.json")
parser.add_argument("--cpu-results", default=CPU_RESULTS,
help="the CPU benchmark's published results file")
args = parser.parse_args()
with open(args.results, encoding="utf-8") as fh:
payload = json.load(fh)
published = None
if os.path.exists(args.cpu_results):
with open(args.cpu_results, encoding="utf-8") as fh:
published = json.load(fh)
text = render(payload)
text = render(payload, cpu_reference(published))
sys.stdout.write(text + "\n")
summary = os.environ.get("GITHUB_STEP_SUMMARY")
benchmarks/gpu/test_bench.py
+40 -11
View File
@@ -99,7 +99,7 @@ def _entry(**changes):
"tier_before": {"pro": True},
"tier_after": {"pro": True},
"parity": {"status": "exact"},
"unstable_reps": {"gpu": 0, "cpu": 0},
"unstable_reps": 0,
}
entry.update(changes)
return entry
@@ -110,7 +110,7 @@ def _entry(**changes):
"fell back to the CPU"),
({"tier_after": {"pro": False}}, "tier changed"),
({"parity": {"status": "failed"}}, "differ"),
({"unstable_reps": {"gpu": 1, "cpu": 0}}, "did not reproduce"),
({"unstable_reps": 1}, "did not reproduce"),
])
def test_verdict_fails(changes, reason):
status, why = bench.verdict(_entry(**changes))
@@ -127,30 +127,59 @@ SAMPLE = {
"status": "exact",
"reps": 5,
"environment": {
"platform": "Modal L4, CPU side on 4 reserved cores",
"cpu": "AMD EPYC", "cpu_threads": 4,
"platform": "Modal L4",
"cpu": "unknown", "cpu_threads": 4,
"gpus": [{"name": "NVIDIA L4", "driver_version": "570.0",
"memory.total": "23034 MiB"}],
"versions": {"manifoldbt": "0.28.1"},
},
"sweeps": [
{"bars": 20000, "combos": 5000, "status": "exact", "reason": None,
{"bars": 20000, "combos": 5000, "data_digest": "554df142406ad3f0",
"status": "exact", "reason": None,
"parity": {"status": "exact", "identical_cells": 5000},
"timings": {"seconds": {"gpu": 0.02, "cpu": 0.2}, "speedup": 10.0,
"per_combo_us": {"gpu": 4.0, "cpu": 40.0},
"first_call_s": {"gpu": 0.03, "cpu": 0.21}, "noisy": False}},
"timings": {"seconds": 0.02, "per_combo_us": 4.0, "first_call_s": 0.03,
"noisy": False}},
{"bars": 200000, "requested_combos": 10000, "status": "failed",
"reason": "RuntimeError: out of memory"},
],
}
def test_report_renders_both_kinds_of_row():
text = report.render(SAMPLE)
CPU_PUBLISHED = {
"run_url": "https://github.com/manifoldbt/manifoldbt/actions/runs/1",
"sweeps": {
"environment": {"versions": {"manifoldbt": "0.28.1"}, "logical_cores": 4},
"sweeps": [
{"bars": 20000, "combos": 5000, "data_digest": "554df142406ad3f0",
"timings": {"seconds": {"manifoldbt": 0.228}}},
# Same grid, other data: must not be matched.
{"bars": 200000, "combos": 10000, "data_digest": "other",
"timings": {"seconds": {"manifoldbt": 3.5}}},
],
},
}
def test_report_reads_the_cpu_column_from_the_cpu_job():
text = report.render(SAMPLE, report.cpu_reference(CPU_PUBLISHED))
assert "NVIDIA L4" in text
assert "| 20,000 | 5,000 | 20.0 ms | 0.200 s | x10.0 |" in text
assert "| 20,000 | 5,000 | 20.0 ms | 4.00 us | 30.0 ms | 0.228 s | x11.4 |" in text
assert "5,000/5,000 bit-identical" in text
assert "failed: RuntimeError: out of memory" in text
assert "4-vCPU GitHub runner" in text
def test_report_gives_no_ratio_across_versions():
older = json.loads(json.dumps(CPU_PUBLISHED))
older["sweeps"]["environment"]["versions"]["manifoldbt"] = "0.27.2"
text = report.render(SAMPLE, report.cpu_reference(older))
assert "| 0.228 s | |" in text
assert "no ratio" in text
def test_report_without_a_cpu_file():
text = report.render(SAMPLE, report.cpu_reference(None))
assert "| 20,000 | 5,000 | 20.0 ms |" in text
def test_publish_refuses_a_failed_result(tmp_path):
benchmarks/vs_vectorbt/README.md
+1 -1
View File
@@ -552,7 +552,7 @@ same point measured x13 on a four-vCPU cloud runner and x32 on two fast desktop
cores; the CI numbers are the conservative ones, and they are the ones published.
These runners have no GPU, so this job measures the CPU path of a sweep only. The
same three points on a GPU, with the CPU of that machine beside it, are in
same three points timed on a GPU, and read against this job's numbers, are in
[`../gpu`](../gpu/README.md).
### NautilusTrader and the grid